Site Administrator Swing Trader Data Scientist & DevOps
Manta, Ecuador
Experience: Advanced
Platform: Custom solution
Trading: Futures & Crypto
Posts: 50,006 since Jun 2009
Thanks: 32,469 given,
98,292
received
I am glad to report Dexter had a better day today. So did I.
I am still struggling to deal with the data center fire. It had a lot of ripple effects, including our cloud drive which was hosted on Nextcloud on those servers. So, I've been migrating everything from the off-site backup to Wasabi S3 (Amazon S3 clone, far cheaper).
For those that care, I've tried s3fs which was unbearably slow. Then I found goofys, and it's quite good for reasonable read/write at any given time, and many factors faster than s3fs.
But in the end, for our servers at least, I landed on s3backer. This project is fantastic. It lets me create a filesystem on top of it -- the author is a big fan of ReiserFS, which I've never used personally (author went to jail for child porn, as I recall, but supposedly wrote one hell of a filesystem). I tried XFS on top first and it just kept throwing errors under heavy load, but under Reiser it works perfectly.
s3backer lets me do a lot of granular control and caching, so the speed is really fantastic. That solved the server issue, so they can backup to Wasabi S3 now instead of the off-site physical server. This will make pulling the files back down during a full off-site restore much faster. It also allows me to keep the off-site in sync, instead of daily, I think I can do hourly once the new server is deployed.
For my home workstation, Windows, however --- lots of problems. I got rid of Google Drive a while back, and like I said, was running Nextcloud. I had a rather complex setup involving WebDAV replication across multiple Synology servers, so that basically any edit was pushed across three servers within seconds. And the important documents in my "Documents" library (Windows) were all on my local NVMe, for fastest speed inside my IDE (Atom).
After trying several alternatives, to eliminate Nextcloud this time around, I finally landed on Mountain Duck. Mountain Duck is actually a nifty app that lets you mount many different protocols as a local Windows drive, and it offers caching as well where needed (like when traveling).
Of course, everything takes time. Having to push terabytes of data at 20MB/sec takes a while. And I keep running into small issues, like permissions needing to be reset on the bucket (user policies), or the modified date not being correct on the target, etc. Tonight is night four, I think, of my attempt to restart the entire copy from off-site to Wasabi.
I should mention, "off-site" is my house across two Synology servers with RAID 6 and around 300TB of capacity.
Also, throw in our new Webinar Framework, and things continue to get more interesting. What I am building allows me to automatically download the source Zoom files upon webinar completion, so we get all the different raw MP4 views (like webcam only, audio only, share screen only, gallery only, and combined). My code auto grabs it, and stores it (using Goofys) to a bucket that stores our webinars. This is so I can share everything easily to our marketing group, like Shane. It's important also so that eventually when a webinar is ended, we can auto-publish to our recording gallery (YouTube for example) very quickly.
Anyway, I've been making a lot of these kinds of changes and coding and work behind the scenes lately. But I hope it shows in the final product, that our community stands out far and above others with all of our extra features, especially the webinars.
