User talk:Fæ

for older archives, see User talk:Fæ/2021, User talk:Fæ/2020, User talk:Fæ/2019, et seq.

Old projects

• Exemplars for historical documents and maps projects
• 
  Scan released last year at the Internet Archive
• 
  New upload of 1887 address book for Riga, Latvia. (800mb).
• 
  Example map, (created 1774) high resolution map of Africa, approx 373 megapixels in size, similar to a 5 foot square printed poster.

I had a look back at the IA upload project and realized that none of these scripts run because of Python, Pywikibot and internetarchive changes. It turns out I find it quite difficult to remember almost anything about these projects too. So don't be surprised if I'm testing it out and there are flaws. I'll do my best to repair any oddities. I'm seeing '+99' notices from my account which doesn't clear, could be a wm bug for big numbers, so I might not notice changes. It's not deliberate. For the moment please don't ask me to take on large projects, I'd rather pace myself at 'slow'. Fæ (talk) 13:07, 2 June 2026 (UTC)

Thanks, and welcome back!   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 08:11, 3 June 2026 (UTC)

Another new "feature" of the Wikimedia API is ratelimiting. Added a couple of slow down precautions including slapping down multiprocessing, but it's definitely dogging uploads despite being visible to the API as an established user. It may be necessary to revisit the throttling system rather than bumping into it. It's sad this creates extra work for volunteers.

For the first time the queries found restricted items at Internet Archive like 1826histoirenumismatiquedelare, where the Washington University appears to be claiming copyright in a 200 year old publication. Good grief, I hope this is not a trend that IA is tolerating. --Fæ (talk) 07:09, 5 June 2026 (UTC)

I have updated a process for finding the text of Google cover pages in the recent uploads. The 'pending' queue is at Category:OCR detected cover page. Yet to update the page removal process. There's no hurry and I will get to this slowly. --Fæ (talk) 06:29, 8 June 2026 (UTC)

Note that larger files, like the 800mb PDF transcluded, are now possible thanks to limits changing. The larger files might cause things to break, sometimes in predictable ways like the SHA check from mediawiki taking some time to process and be available on the system. Behind the scenes, these uploads do not behave well and invariably fail to report back that they are successfully uploaded. Fæ (talk) 14:26, 17 June 2026 (UTC)

Extra note about maps - though there is a fairly quick upload of jpegs of maps to Category:David_Rumsey_Historical_Map_Collection, which may take a couple of days, there is a much, much slower process to recover high resolution versions. A rough estimate is that this could take 2 or perhaps 4 months, partly due to the new WMF API throttling limits more than processing time. --Fæ (talk) 12:44, 20 June 2026 (UTC)

@Fæ: Couldn't you technically request permission for the upscale to run on Faebot so it is exempt from the API rate limits? --Nintendofan885^{T&Cs apply} 13:30, 20 June 2026 (UTC)

From what is available to easily read about it, probably not. Faebot has a bot flag, but a brief experiment shows that the API throttle is using IP address. Consequently the map uploads being mentioned here are being forced to sleep 90s before the upload is allowed to complete and then there's another 90s for the filepage to be updated with formatting or a category. These events are in a multiprocessing queue but it makes no difference for the API whether it's a bot or not, or whether Pywikibot can set a bot flag for the action.

Keep in mind, I'm trying not to spend hours at a keyboard, so not looking for complex extra volunteer work or phabricator requests for a fairly modest 50,000 files. Fæ (talk) 13:41, 20 June 2026 (UTC)

Faebot has got running on the somewhat revised toolforge, so the mass link fixing has been moved to the WMF servers and that at least is 6x faster as a result and reduced local processing by about 7%. Other stuff might move there eventually. Fæ (talk) 14:50, 21 June 2026 (UTC)

Fresh eye this morning, and the map upgrading is running around 4x faster locally and probably is obeying the new WMF API throttling rules. The run might be 6 weeks rather than 4 months, which is fine. Fæ (talk) 06:39, 22 June 2026 (UTC)

Welcome back

As you probably have worked out, I've been following this talk page, checking whether DRs of your uploads are valid or not, and responding to the DR with a rationale if I think something should be kept. May I assume you are sufficiently "back" that I can let go of monitoring that?

Welcome back, in any case. - Jmabel ! talk 14:06, 3 June 2026 (UTC)

If you wish, though I'll probably continue to ignore most DRs and let others chip in. After your first million images, the DRs that matter are ones that are a meaningful case study for thousands of others and could be collated with some automated method.

I have noticed your patient work and very much appreciate it! Fæ (talk) 15:21, 3 June 2026 (UTC)

• Yes, welcome back! So glad to see this. Krok6kola (talk) 16:06, 3 June 2026 (UTC)
• It's been almost 5 years. I hope you had a good time! Welcome back. - Alexis Jazz ^{ping plz} 10:03, 5 June 2026 (UTC)

@Fæ Yes, happy to see you here again :) --PantheraLeo1359531 😺 (talk) 09:32, 8 June 2026 (UTC)

Good to see you again, also from me. -- Deadstar (msg) 10:27, 9 June 2026 (UTC)

Marvellous to see you editing again. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 15:15, 9 June 2026 (UTC)

Welcome back!

Russian Empire 1883 genealogy interest, 1.16gb, mediawiki struggles at this size.

Looks like you were gone for a bit. Are you working on any new large chunks of uploads? RAN (talk) 21:44, 17 June 2026 (UTC)

Slightly updated the internetarchive PDF uploading, so these are being refreshed:

Category:Scans from University of Toronto

Category:Old books from American Libraries - this in particular needs contributors to surf the content and imagine categories which can better break up the 150,000 books.

Category:David Rumsey Historical Map Collection - new content but a few thousand jpegs of interesting maps. Some review will be needed as the source library curators may have presumed their collections are public domain by age when it can be more complicated.

Category:Images dezoomed by Fæ is being ever so slowly populated by overwriting the small versions from IA. This is probably why this was not done years ago. Hopefully the maps will not be moved before this dezooming task finishes in several days time.

Category:Books in the Prelinger Library

Category:Genealogy books from the Internet Archive modest number of IA books under this collection but some are eye wateringly large, one has been having an upload attempt of over an hour for a 1,781.2 MB PDF; it may continue to fail though several that succeeded are at the 1gb size.

Quietly restarted the slow job of trimming Google cover pages off pdfs which is harder and slower on local processing. Also been experimenting with nccommons, behind the scenes that's been the ghastly issue of getting timedtext to work on videos and battling anti-bot tools for a site that in theory wants you to take their publications.

There is plenty of gnomic work, so for the moment not looking for anything controversial or technically too challenging. Little projects that do not mean sitting at a keyboard for more than an hour at a time are best. Fæ (talk) 05:20, 18 June 2026 (UTC)

• Are we ever going to get something like "search inside this book" when you are at the pdf page, like we get at Google Books? The only time is see the ASCII text is when I do a general search for all of Commons and I see a snippet, or when the book is transcribed at Wikisource. --RAN (talk) 18:59, 18 June 2026 (UTC)

  The search can be used to find PDF contents matches and is a quick way to find prospective matches if doing copyright or cover page statement testing. For example ("This is a digital copy of a book that was preserved for generations" filetype:pdf intitle:IA) will find PDFs with google cover pages. However there's no way of using the mediawiki to tell you what pages the text is on, or do anything else really smart with it. Well no easy designed way; vaguely remember doing something smarter but can't remember how that worked.

A solution for the IA uploads alone would be to navigate from the commons IA upload, back to IA and then interrogate either the djvu.txt file there, or the xml version. If one needed to filter through, say, 100,000 files, that would be way more efficient than having to download each one and run it through another OCR. Fæ (talk) 19:13, 18 June 2026 (UTC)

There is another way of using a wikimedia tool to do it, but not leaving breadcrumbs for it, as if overused it might get switched off. It solves specific issues of examining a page, but is not intended for a million entire books. --Fæ (talk) 13:15, 27 June 2026 (UTC)

+1. Very good to see you back. Jheald (talk) 13:10, 27 June 2026 (UTC)

File:The art of midwifery improv'd Fleuron T115049-2.png

File:The art of midwifery improv'd Fleuron T115049-2.png (edit|talk|history|links|watch|logs)
Commons:Deletion requests/File:The art of midwifery improv'd Fleuron T115049-2.png Jmabel ! talk 23:58, 19 June 2026 (UTC)

File:Mapillary 20H15M52S000 Collier County us (gAS1T8MW4tmO7uqzG6kKvr, 1165545434717956) (cubanoboi) 2024-08-10.jpg

This seems to be a part of a mass upload of Mapillary images.

As these were being uploaded I'd been adding | other_fields = {{information field|Academic context|Geospatial street-level imagery from external site.}}, so these were not seen as random indiscriminaate uploads of 'random' locations and verges.

However the uploads seem to be faster than I can cope with manually. Any chance of seeing if something like Faebot is able to do the additions as the files are uploaded? Thanks. ShakespeareFan00 (talk) 20:00, 20 June 2026 (UTC)

@DaxServer: Would this be a suitable minor addition to the curator, or is it left for others to tack it on if wanted? Fæ (talk) 20:24, 20 June 2026 (UTC)

@Fæ @ShakespeareFan00  Done File:Mapillary (792440271635475, tulzukst7vufhdo1e4z60f) (b4sti4n) 2017-06-24 13H44M25S000.jpg Thanks! -- DaxServer (talk) 11:55, 21 June 2026 (UTC)

for this upload- Thanks. The hope was that the addition could made to related uploads. These are in scope, but given some patrollers... I felt it was reasonable to indicate why they were in fact not just random verge-side images. ShakespeareFan00 (talk) 14:25, 21 June 2026 (UTC)

File:100th Anniversary PNE Parade (4913329591).jpg

File:100th Anniversary PNE Parade (4913329591).jpg (edit|talk|history|links|watch|logs)
Commons:Deletion requests/File:100th Anniversary PNE Parade (4913329591).jpg A1Cafel (talk) 17:44, 21 June 2026 (UTC)

File:The story of the Malakand field force - an episode of frontier war (IA storyofmalakandf00chur).pdf

File:The story of the Malakand field force - an episode of frontier war (IA storyofmalakandf00chur).pdf (edit|talk|history|links|watch|logs)
Commons:Deletion requests/File:The story of the Malakand field force - an episode of frontier war (IA storyofmalakandf00chur).pdf Nighfidelity (talk) 12:51, 24 June 2026 (UTC)

File:The book of knowledge; treating of the wisdom of the ancients Fleuron T083875-5.png

Speedy notice removed.

File:The book of knowledge; treating of the wisdom of the ancients Fleuron T083875-5.png (edit|talk|history|links|watch|logs) Not a fleuron, bad scan of a bit of text.

Action created by Deadstar (talk · contribs · logs · block log) 08:06, 25 June 2026 (UTC)

Category:LSE Commons, events

Speedy notice unclear purpose. Rose Abrams (talk) 13:02, 26 June 2026 (UTC)

File:The Dominion Monarch at a wharf in Wellington, 1940s (3377606529).jpg

File:The Dominion Monarch at a wharf in Wellington, 1940s (3377606529).jpg (edit|talk|history|links|watch|logs)
Commons:Deletion requests/File:The Dominion Monarch at a wharf in Wellington, 1940s (3377606529).jpg Traumnovelle (talk) 04:05, 28 June 2026 (UTC)

File:Professional photos (16398186965).jpg

File:Professional photos (16398186965).jpg (edit|talk|history|links|watch|logs)
Commons:Deletion requests/File:Professional photos (16398186965).jpg JaydenChao (talk) 04:52, 28 June 2026 (UTC)

File:View inside powerhouse from entrance door - Titan One Missile Complex 2A, .3 miles west of 129 Road and 1.5 miles north of County Line Road, Aurora, Adams County, CO HAER CO-89-12.tif

File:View inside powerhouse from entrance door - Titan One Missile Complex 2A, .3 miles west of 129 Road and 1.5 miles north of County Line Road, Aurora, Adams County, CO HAER CO-89-12.tif (edit|talk|history|links|watch|logs)
Commons:Deletion requests/File:View inside powerhouse from entrance door - Titan One Missile Complex 2A, .3 miles west of 129 Road and 1.5 miles north of County Line Road, Aurora, Adams County, CO HAER CO-89-12.tif JopkeB (talk) 06:54, 29 June 2026 (UTC)

Purging thumbnails

Due to old glitches with uploads and other inscrutable problems with WMF operations, some of the very large categories of upload projects have multiple un-thumbnailed files with apparent 0x0 dimensions. These are fairly easy to search and repair individually by purging the cache, forcing a thumbnail refresh, but mass finding and repairing does not have an easy tool for non sysop users. Here's my current pywikibot based solution for fast purging given a category name from a terminal prompt. You just need to know how to install pywikibot, or have a toolforge session.

• User:Fæ/code/thepurge

Fair warning, this is deliberately very fast but API throttling *probably* does not count purge actions as they are non-edits. Worth noting that in the example of incategory:Old_books_from_American_Libraries filewidth:<1 before the programme there were 2,433 files with zero width and after running (a couple of minutes duration) the same files show a correct thumbnail but the search results on-wiki incorrectly show the same number of 0x0 thumbnails. Presumably the database lag takes several minutes or maybe hours. Fæ (talk) 08:35, 29 June 2026 (UTC)

Removing "Historic, Archive Document" cover pages

This cover page should be removed in a few weeks time.

The refresh of Biodiversity Heritage Library documents from the internet archive has highlighted again the old scans with an extra unattractive cover page with this warning notice. These cover pages were not used at the BHL original archive site, but seem to have been added by the scanners for the IA transfer. When the links to IA are followed, interestingly the cover page is skipped and the document is displayed from the real first page. Policies on these warning notices also seem to have changed and it probably fell out of fashion 9 years ago. Faebot is running through what will be around 100,000 of these pdfs to attempt to trim these manually added cover pages which vary in colour, size and even text between documents. It carefully tries to see if there is a blank second page, often faded to a cream colour, to trim off based on shade and the absence of text, but due to caution may leave these in place in around 25% of cases. The end result is that categories like Category:Henry G. Gilbert Nursery and Seed Trade Catalog Collection/1929 are made fantastically more attractive to volunteers and the pdfs far more likely to be used in articles or elsewhere as is.

The task is attempting a batch to process daily via toolforge in alphabetical order and may take, say, 6 weeks. Fæ (talk) 09:03, 29 June 2026 (UTC)