Recent comments posted to this site:

Is it possible to edit comments on the branchable wiki? I realized there was a sentence I failed to finish when posting this comment and I'd love to go back and finish the thought. The "Edit" button at the top of the page lets me edit the content of the page, but not any of the comments.

I tried cloning the wiki, editing the file corresponding to my comment, and then pushing, but the push was rejected (the changes were in doc tree so I expected it to be accepted, but perhaps comments are more locked down).

Comment by Dan Thu Aug 11 14:28:49 2022

If you add a file to your repo first via addurl --fast, it writes the filename as a symlink to a file that incorporates the URL, rather than the file hash. This is expected, since git-annex can't know the file hash until it's actually downloaded the file.

If you then git annex get that file, it downloads the file to the path that uses the URL. Is the hash ever recorded for these files? If you were to drop and re-download the file, would git-annex accept a different file?

Comment by tomdhunt Thu Jul 21 14:00:05 2022

Is the hash ever recorded for these files? If you were to drop and re-download the file, would git-annex accept a different file?

Hash is not recorded, but file size is. You can disable the size check with --relaxed. See using the web as a special remote. After git-annex-getting the file, you can use git-annex-migrate to record it under a new checksum-based hash, then use git-annex-unused to find and remove the old key.

Sometimes you can get the hash without downloading the file, e.g. if the hash is stored next to the file at http://my/file.md5, or if the file is stored in the Google Cloud. Then you can use the plumbing commands git-annex-registerurl to associate the checksum-based key with the URL, and git-annex-setpresentkey to record the key's presence in the (web) remote.

Related discussion: alternate keys for same content

Comment by Ilya_Shlyakhter Thu Jul 21 14:00:05 2022
Thanks for taking the time to direct me, Joey. I usually find myself getting lost in this wiki amongst all the old notes and worklogs and documentation about old features, so sometimes someone coming along with a sign post is a very kind help :)
Comment by nick.guenther Thu Jul 21 14:00:05 2022

starting git annex webapp fails for me with (Android 12, CalyxOS):

CANNOT LINK EXECUTABLE "/system/bin/app_process": library "libnativeloader.so" not found: needed by main executable
failed to start web browser

The webapp still runs and I can connect the browser by copypaste the link.

instead trying to open the browser natively i have a bit more success with Termux:API by

termux-open-url 'http://127.0.0.1:42005/?auth=...

This seems to be the right thing to do, unfortunally for some unknown reason it fails sometimes too, but I think thats an issue within termux which may become fixed eventually.

Comment by cehteh Thu Jul 21 14:00:05 2022

It seems that for me at least, OSX installations performed with brew install git-annex do not include man pages, e.g., if I subsequently run man git-annex I get the error No manual entry for git-annex. This isn't an enormous bother as I can instead browse documentation as needed on the website, but it's a headache when I'm working offline and even when not it can be irksome to have to leave the terminal.

Below I've pasted the relevant lines that govern how homebrew performs the installation (or in practice, how the "bottle" is built as part of a CI process, and which bottle is typically "poured" by end users rather than compiling locally).

  def install
    system "cabal", "v2-update"
    system "cabal", "v2-install", *std_cabal_v2_args,
                    "--flags=+S3"
    bin.install_symlink "git-annex" => "git-annex-shell"
  end

Notably, it uses cabal to do the installation, but it doesn't quite follow any of the cabal-related patterns currently outlined on the "installing from source" documentation. I inspected the bottle that homebrew downloads, and indeed it contains just a handful of files that do not include the man pages, i.e.:

.
├── CHANGELOG
├── COPYRIGHT
├── NEWS
├── README
├── bin
│   ├── git-annex
│   └── git-annex-shell -> git-annex
├── homebrew.git-annex.service
└── homebrew.mxcl.git-annex.plist

1 directory, 8 files

Is there an easy tweak to the cabal-based installation that homebrew currently uses to do its bottling that would include man pages, or would it be a potentially significant undertaking that would require an appreciably different build approach?

Comment by Dan Wed Jul 13 19:21:00 2022

@Dan I would not be surprised if cabal v2-install does not support installing man pages like the old v1-install does. When I tried it, it seemed not to.

The Makefile has an install-mans target that could be used instead. There are several other things that make install does that cabal cannot, including installing completions files, so system integrators (like homebrew) should probably use it when they can.

Comment by joey Wed Jul 13 19:21:00 2022

@g, when you have a problem like this, don't post a comment to a random place. File a bug report!

In this case, someone else has filed a bug: adb remote fails to import treeish

Comment by joey Wed Jul 13 19:21:00 2022

Amazon has deprecated ACLs

A majority of modern use cases in Amazon S3 no longer require the use of ACLs, and we recommend that you disable ACLs except in unusual circumstances where you need to control access for each object individually. With Object Ownership, you can disable ACLs and rely on policies for access control. When you disable ACLs, you can easily maintain a bucket with objects uploaded by different AWS accounts. You, as the bucket owner, own all the objects in the bucket and can manage access to them using policies.

They are encouraging everyone to migrate to bucket policies instead.

Implementation

I've done this for a bucket I run. I wrote and attached this Bucket Policy to it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::BUCKET-NAME",
                "arn:aws:s3:::BUCKET-NAME/*"
            ]
        }
    ]
}

I had to reset the ACLs on that bucket to the default:

  • Bucket owner (your AWS account):
    • Objects:
      • List
      • Write
    • Bucket ACL:
      • Read
      • Write

and with that set Amazon let me also set

  • Object ownership: Bucket owner enforced

git-annex incompatibility

But, attempting to use the new setup failed:

$ git annex copy --to amazon what.nii.gz 
copy what.nii.gz (checking amazon...) (to amazon...) 
41%   8.15 MiB         20 MiB/s 0s
  S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "a6+ieujj4z3Z4P8ooA306DdbGAoxWDiXd6O2ZwjdfapGnuOGPyL5/WQ4UBEytR80FG+5b6xdlsM=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}

32%   6.43 MiB         16 MiB/s 0s 
  S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "bFOgMomROCOes9yI6HZHysQGoZaTbsPI5b7rHjcTI0wA8Yx5Dm1JOky9BvXvpcXxzY1kVt48FRQ=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}

37%   7.37 MiB         21 MiB/s 0s 
  S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "hqd4HRNk5yp3tKJ6yMhcECEpCjBw8qB6oTpKF3PaOsYFeVG0C+dGI06xq3zgmvnPoFUttI040sY=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}

39%   7.81 MiB         21 MiB/s 0s 
  S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "7m7wwG5woSPmICIuXr9QnBOEjUikuyzHSebMLuaNyZMc2Ki2vaqKpU9U+GOTYmR/NzFjOeyxngk=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
failed
git-annex: copy: 1 failed

Workaround

However, this fixed it:

$ git annex enableremote amazon public=no
enableremote amazon ok
(recording state in git...)
$ git annex copy --to amazon what.nii.gz 
copy what.nii.gz (checking amazon...) (to amazon...) 
ok                                

Feature Request

  • If public=yes, instead of trying to set an ACL, first try HEAD on the newly uploaded object without using the AWS_ACCESS_KEY. Only if that fails, fall over to trying to set an ACL. And if you get AccessControlListNotSupported (i.e. the error due to BucketOwnerEnforced), then give a warning that the bucket policy is not configured for public access.
  • Update the docs here to explain how to set up a public bucket policy as recommended by Amazon, and that public=yes will either try to confirm that the bucket policy is public, or will fallback to using ACLs.
Comment by nick.guenther Wed Jul 6 16:28:50 2022

@nick.guenther A comment on this page is really not the place to report a bug like this. See bugs for where to file a bug.

Comment by joey Wed Jul 6 16:28:50 2022