First, I'd like to appologize to everyone for shutting down remotepixel.ca without notice last Friday. This blog post will explain why I had not choice.
Second, I need to re-introduce myself so the following will make more sense. My nane is Vincent Sarago, I'm the `OWNER` (read: `the only person working on it`) of this website. Remotepixel.ca started as PERSONAL PROJECT and will stay a hobby for my cold winter night (winter is quite long in Montreal, QC) and now I have a `real` job that I love and keeps me busy during the week.
Remotepixel is Free... not to everyone
WTF my aws bill is up to 3k$ this month, I’m sorry but I’m shutting down https://t.co/aqc4TNnJoF— Remote Pixel (@RemotePixel) August 31, 2018
Remotepixel is not just a webpage, almost every project I worked on has one or more services (API) associated. I host Remotepixel.ca web pages and services on Amazon Web Services (AWS) platform and last week when I openned the `billing section` I had this nice surprise ☝️.
The bill was around 3 500 $ (4 510 $ CAD) 😱😱😱😱😱.
My first reaction was to log off because I though I logged with my Mapbox credentials, but then the reallity was brutal, this was my own bill. At this point I had no choice but to shut down everything as fast as I could.
While I'm usually happy to support the cost of running remotepixel.ca (~100$/month), this was 3000% more than my usual bill, and way too high for my `happy to give it for free budget`.
Remotepixel was finally popular with thousand of thousand of people using it.
☝️ even with this possibility, AWS services are not that expensive, and google analytics quickly anwser this question: RemotePixel is still a `niche` with around 5000 users per month.
So the web users are not to blame (no huge increate in users or in session duration), but there is something odd the billing is saying.
RemotePixel's services are based on AWS Lambda which is a service that let you run python/node/java scripts without caring for a server (serverless). I'm running AWS Lambda functions in each region where I need to access data (eu-central-1 for Sentinel-2, us-west-2 for Landsat-8), so it's then easy to see if a specific dataset/services was used more.
In the previous screenshots we can see that AWS lambda function in `eu-central-1` (sentinel-2 access) where called 1.5 MILLION times more in august in comparison to july (which was also an expensive month ~180$). Note how the price for the other region didn't change much !!!
But AWS Lambda pricing is cheap. This cost me `only` 290$. 😒😒😒 We still need to explain for 3100 $.
Remember my post about Sentinel-2 S3 bucket moving to `requester-pays` ? Well this is now in place and we don't have to look too far to see the first consequence. If we look at my S3 cost in eu-central-1 the total is around 3000$ 🎉.
480 000 LIST + 950 000 GET request in one month 😱😱😱
Is this my fault ? Mostly
Because Sentinel-2 bucket is in `requester-pays` it means that if you have an application accessing the data running outside the `eu-central-1` region, you'll be charged with the data transfer cost. But RemotePixel services run in `eu-central-1` so it should be costless... Well that's not true and the previous screen shot tells you why. You don't pay for data transfert but you have to pay for every LIST/GET requests.
In general, bucket owners pay for all Amazon S3 storage and data transfer costs associated with their bucket. A bucket owner, however, can configure a bucket to be a Requester Pays bucket. With Requester Pays buckets, the requester instead of the bucket owner pays the cost of the request and the data download from the bucket. The bucket owner always pays the cost of storing data.
This is something I didn't though about, and because some of my services are doing a lot of LIST/GET, event with a normal usage, my AWS bill would have been high. My Search API seems to be a really bad idea now.
Is this my fault ?
Mostly Who's scrapping
The numbers don't lie, remotepixel.ca users didn't cause all of the bill increase but someone has been scrapping my services.
Using some companies API is something, but abusing of a free service and/or scrapping is just not fair. Especially when I always try to publish the code of every service and web page I create.
That's said, the GET/LIST numbers seems really odd and I'll need to deep dive on it (another blog post soon).
TERM OF SERVICE
Since I shut down, I've received a lot of support from different individuals or companies, I might not answered everyone but be sure that I've seen it.
As usual I also received mail from companies complaining about the website/services being down 😒.
So let's be clear again, this is a personal project and:
- Date: September 2018
- Category: RemotePixel