Recent comments posted to this site:
If you add a file to your repo first via addurl --fast
, it writes the filename as a symlink to a file that incorporates the URL, rather than the file hash. This is expected, since git-annex can't know the file hash until it's actually downloaded the file.
If you then git annex get
that file, it downloads the file to the path that uses the URL. Is the hash ever recorded for these files? If you were to drop and re-download the file, would git-annex accept a different file?
Is the hash ever recorded for these files? If you were to drop and re-download the file, would git-annex accept a different file?
Hash is not recorded, but file size is. You can disable the size check with --relaxed
. See using the web as a special remote.
After git-annex-get
ting the file, you can use git-annex-migrate
to record it under a new checksum-based hash, then use git-annex-unused
to find and remove the old key.
Sometimes you can get the hash without downloading the file, e.g. if the hash is stored next to the file at http://my/file.md5
, or if the file is stored in the Google Cloud. Then you can use the plumbing commands git-annex-registerurl
to associate the checksum-based key with the URL, and git-annex-setpresentkey
to record the key's presence in the (web) remote.
Related discussion: alternate keys for same content
starting git annex webapp
fails for me with (Android 12, CalyxOS):
CANNOT LINK EXECUTABLE "/system/bin/app_process": library "libnativeloader.so" not found: needed by main executable
failed to start web browser
The webapp still runs and I can connect the browser by copypaste the link.
instead trying to open the browser natively i have a bit more success with Termux:API by
termux-open-url 'http://127.0.0.1:42005/?auth=...
This seems to be the right thing to do, unfortunally for some unknown reason it fails sometimes too, but I think thats an issue within termux which may become fixed eventually.
It seems that for me at least, OSX installations performed with brew install git-annex
do not include man pages, e.g., if I subsequently run man git-annex
I get the error No manual entry for git-annex
. This isn't an enormous bother as I can instead browse documentation as needed on the website, but it's a headache when I'm working offline and even when not it can be irksome to have to leave the terminal.
Below I've pasted the relevant lines that govern how homebrew performs the installation (or in practice, how the "bottle" is built as part of a CI process, and which bottle is typically "poured" by end users rather than compiling locally).
def install
system "cabal", "v2-update"
system "cabal", "v2-install", *std_cabal_v2_args,
"--flags=+S3"
bin.install_symlink "git-annex" => "git-annex-shell"
end
Notably, it uses cabal
to do the installation, but it doesn't quite follow any of the cabal-related patterns currently outlined on the "installing from source" documentation. I inspected the bottle that homebrew downloads, and indeed it contains just a handful of files that do not include the man pages, i.e.:
.
├── CHANGELOG
├── COPYRIGHT
├── NEWS
├── README
├── bin
│ ├── git-annex
│ └── git-annex-shell -> git-annex
├── homebrew.git-annex.service
└── homebrew.mxcl.git-annex.plist
1 directory, 8 files
Is there an easy tweak to the cabal-based installation that homebrew currently uses to do its bottling that would include man pages, or would it be a potentially significant undertaking that would require an appreciably different build approach?
@Dan I would not be surprised if cabal v2-install does not support installing man pages like the old v1-install does. When I tried it, it seemed not to.
The Makefile has an install-mans
target
that could be used instead. There are several other things
that make install
does that cabal cannot, including installing
completions files, so system integrators (like homebrew)
should probably use it when they can.
@g, when you have a problem like this, don't post a comment to a random place. File a bug report!
In this case, someone else has filed a bug: adb remote fails to import treeish
Amazon has deprecated ACLs
A majority of modern use cases in Amazon S3 no longer require the use of ACLs, and we recommend that you disable ACLs except in unusual circumstances where you need to control access for each object individually. With Object Ownership, you can disable ACLs and rely on policies for access control. When you disable ACLs, you can easily maintain a bucket with objects uploaded by different AWS accounts. You, as the bucket owner, own all the objects in the bucket and can manage access to them using policies.
They are encouraging everyone to migrate to bucket policies instead.
Implementation
I've done this for a bucket I run. I wrote and attached this Bucket Policy to it:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::BUCKET-NAME",
"arn:aws:s3:::BUCKET-NAME/*"
]
}
]
}
I had to reset the ACLs on that bucket to the default:
- Bucket owner (your AWS account):
- Objects:
- List
- Write
- Bucket ACL:
- Read
- Write
- Objects:
and with that set Amazon let me also set
- Object ownership: Bucket owner enforced
git-annex incompatibility
But, attempting to use the new setup failed:
$ git annex copy --to amazon what.nii.gz
copy what.nii.gz (checking amazon...) (to amazon...)
41% 8.15 MiB 20 MiB/s 0s
S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "a6+ieujj4z3Z4P8ooA306DdbGAoxWDiXd6O2ZwjdfapGnuOGPyL5/WQ4UBEytR80FG+5b6xdlsM=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
32% 6.43 MiB 16 MiB/s 0s
S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "bFOgMomROCOes9yI6HZHysQGoZaTbsPI5b7rHjcTI0wA8Yx5Dm1JOky9BvXvpcXxzY1kVt48FRQ=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
37% 7.37 MiB 21 MiB/s 0s
S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "hqd4HRNk5yp3tKJ6yMhcECEpCjBw8qB6oTpKF3PaOsYFeVG0C+dGI06xq3zgmvnPoFUttI040sY=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
39% 7.81 MiB 21 MiB/s 0s
S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "AccessControlListNotSupported", s3ErrorMessage = "The bucket does not allow ACLs", s3ErrorResource = Nothing, s3ErrorHostId = Just "7m7wwG5woSPmICIuXr9QnBOEjUikuyzHSebMLuaNyZMc2Ki2vaqKpU9U+GOTYmR/NzFjOeyxngk=", s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}
failed
git-annex: copy: 1 failed
Workaround
However, this fixed it:
$ git annex enableremote amazon public=no
enableremote amazon ok
(recording state in git...)
$ git annex copy --to amazon what.nii.gz
copy what.nii.gz (checking amazon...) (to amazon...)
ok
Feature Request
- If public=yes, instead of trying to set an ACL, first try
HEAD
on the newly uploaded object without using theAWS_ACCESS_KEY
. Only if that fails, fall over to trying to set an ACL. And if you getAccessControlListNotSupported
(i.e. the error due toBucketOwnerEnforced
), then give a warning that the bucket policy is not configured for public access. - Update the docs here to explain how to set up a public bucket policy as recommended by Amazon, and that
public=yes
will either try to confirm that the bucket policy is public, or will fallback to using ACLs.
@nick.guenther A comment on this page is really not the place to report a bug like this. See bugs for where to file a bug.
Is it possible to edit comments on the branchable wiki? I realized there was a sentence I failed to finish when posting this comment and I'd love to go back and finish the thought. The "Edit" button at the top of the page lets me edit the content of the page, but not any of the comments.
I tried cloning the wiki, editing the file corresponding to my comment, and then pushing, but the push was rejected (the changes were in doc tree so I expected it to be accepted, but perhaps comments are more locked down).