aws s3/ec2/emr tips+tricks

Discussion in 'Tech Heads' started by Agrul, Nov 13, 2017.

  1. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    any1 got any

    gimme em
     
  2. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    that's not too broad at all.

    next post from agrul:
    "computers = any tipskthx"
     
  3. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    it's intentionally broad u shitty IT person

    do u have tips for saving money on storage / compute?

    tips for saving money during dev before moving to a pricier node/cluster for real work?

    great articles on the differences between EBS and nodes that come with standard SSD

    observations on what aws nodes cannot do? (eg recvently discovered they doesnt seem to have any kind of graphical terminal available, 2 the point that it is apparently meaningless to install python's tkinter on them)

    do u have fancy spot pricing rules? are ur spot pricing rules the next chain linkz?

    what do u use s3 / ec2 / emr for? anything interesting ppl wouldnt normally think of?

    gimme ur secrets all of them
     
  4. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    this was prompted by my recently:

    A) spinning up a p2.xlarge gpu ec2 node, doing some work on it

    waiting 2 days

    B) finding my limit for p2.xlarge nodes had suddenly dropped to 0

    C) ???

    D) talk 2 aws ppl

    E) get non-0 limit

    F) no idea why i was able to run one in A)

    anyway this made me think there are probably many quirks to aws i am unaware of so talk about all of them
     
  5. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    there are so many quirks it would be hard to know where 2 start. i'll try to blast some answers 2 ur questions shortly though.

    also, i've not used a p2.xl gpu node but i'm going to guess it's going to cost about 1 arm + 1 leg 2 run per/hr
     
  6. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    big agrul w/the big budget
     
  7. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    big agrul = government bloat
     
  8. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    drain agrul
     
  9. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    use windows desktops, not aws - make america great again
     
  10. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    p2.xlarge isn't too expensive, still <$1 / hr typically, and with spot pricing i have gotten them for like $0.2-0.4 / hr mostly

    i mean it would be expensive to run one for a month straight 4 sure, but that's not my plan u know

    but yah the cost is why the 0 limit apparently

    to protect us from ourselves + scammers, altho still doesnt explain why i was able to create one originally
     
  11. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    lookin at my graph

    upload_2017-11-13_20-16-25.png

    it would appear in fact that i got 1-2 hrs of p2.xlarge for either $0.17 or $0.34

    that seems p reasonable to me
     
  12. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    dat $0.47 is way wrong tho imma make p2 do some wurk this weekend
     
  13. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    not rly, except to have automation to shut stuff down if it's not being used. that's a rly obvious tip that in practice is an enormous amount of work for tech shops that are not used to automation.

    also, watch ur bills like a hawk. i can't think of any company i have worked for that did not get sticker-shock at least once because they just didn't realize how costs would skyrocket until bezos already owned their souls.

    again nothing specific, just the obvious - for functional dev don't use full size instances if u don't need it.

    honestly, AWS docs are generally excellent and it would be hard to do better than just reading what amazon already publishes. that said, i'm not sure if ur differentiating between EBS and ephemeral SSD. basically ephemeral anything just needs to be considered during achitecture stage. probably one of the most common newbie mistakes ppl make is writing stuff to some ephemeral local disk then crying after they reboot and lose all their shit.

    i've heard that nesting virtualization layers on EC2 can be problematic/impossible depending on what ur trying to do. i can't imagine you'd be doing that though so not relevant probably. i can't rly think of much else outside your own discovery.

    well, actually, i'll say that observing network traffic and troubleshooting network oddities is a place where AWS is a pain, because in-house you can get switch/router logs and counters and stuff and in AWS none of that stuff is really exposed, so it can be a bit hairy if you suffer weird/random network probs.

    another thing that's tough at scale (network-related) is knowing what public IPs ur AWS stuff is going to use. this is irrelevant for most ppl, but in cases where you always need to know your outbound IP from AWS (for example if u need to IP whitelist that traffic), those ips change all the time. you can work around this using NAT gateways and assigned elastic IPs, but it's best to do this up-front from the beginning if u know ur gonna need it. This is just for ec2 shit. I don't even know if it's doable w/the other 2340932409203234 services AWS offers.

    i personally have not dabbled in this, but yes ppl do make a science out of spot pricing shenanigans. i've worked at a place that had a whole department soley dedicated to gaming AWS spot pricing to save money, and i'm sure despite the cost of those engineers it was probably well worth the investment.

    ec2 runs TZT. been p good so far.

    s3 has been used for all the things in my past 4 workplaces. too many use cases to describe. make sure 2 secure ur buckets or else you'll get pwned.

    emr seems useful but for some reason we decided to go w/databricks @ work... that's been pretty solid (because i think a lot of the apache haderp/spark original developers were the ones who founded databricks)
     
  14. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    my understanding is that databricks does a better job optimizing spark jobs than native spark in emr (the databricks ppl claim rather dramatic speedups on some jobs, something like an order of magnitude or even two iirc); i think i would rather be working in it. emr has the advantage that it is lazier to setup though (very important in comically bureaucratic work environments)

    does databricks have as nice a range of machine images available? the rather complex software (e.g. tensorflow and theano and torch and ... u know all the gay machine learning software; very strangely tho they dont have good AMIs for operations research or standard stats etc) available pre-installed on ec2 AMIs is pretty impressive. the reason i started dicking around with this p2.xlarge for example is because i'm giving a talk at the end of this month about convolutional neural nets, and i wanted access to a gpu to do that, but i didn't want to try to get theano or any of the other big data frameworks functioning on my home pc (for a # of reasons, mainly: currently wnidows pc & 2 lazy to setup another dual boot right now but windows sux at real good softwares, 2ndarily bc i have a nice-ass gpu but it is driving my nice-ass monitor and running compute off the same gpu driving ur monitor is itself a world of headaches, especially in windows)

    ec2 runs tzt? you have like an ec2 node running that acts as tzt's server? what are the monthly costs like on that?

    do you have a preferred way of spinning up short-lived nodes/storage and moving homegrown software to them? for the cnn talk i'm giving i've settled on moving shit to a git repo (including data, since i picked a small example that's still shy of 1GB, though not by much) and cloning to the ec2 node whenever i spin it up. previously i wrote a python script to do essentially the same thing but the git workflow seems lazier and more fuckup-proof
     
  15. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    how come u phishin me utumno

    upload_2017-11-13_20-55-48.png
     
  16. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    i remember a p wide range but am too tired/lazy 2 look up now. may respond to this later in the week when i'm looking at our shizzy and already logged in

    yes. ec2 node running xenforo - a t2 medium, which is admittedly overkill for our needs but i wanted 2 be able to add stuff if needed + being new to xenforo i didn't know if it was as lightweight as promised. also using ebs storage (80gb gp2 volume), which seems plenty - 240 iops which can burst to 3k if needed. this is shitloads better than the old host where we would frequently max out iops and the mysql db would slow 2 a crawl. price? i also have a t2.nano bastion host so i'm not 100% sure but i'm guessing it's ~$30/mo or so just for tzt server.

    i do not. i was lazy and did all the setup by clicking buttons and shit in aws console, and i told myself i would either use cloudformation or terraform + ansible to automate all the things, but never got around to it. i hear cloudformation is probably the ezmode way of doing automated spinups then use ansible 4 config-mgmt on ur new nodes. i guess even easier/lazier is to just save your Known Good Node as an AMI (if we're strictly speaking EC2 only here) and then spin those up/down as needed and not even bother w/config mgmt or host automation beyond start/stop scripts - but that's obviously more inflexible since every time u need to make changes you'd need to generate new ami and fuck that.
     
  17. Utumno

    Utumno Administrator Staff Member

    Post Count:
    36,435
    on my to-do-list-but-will-probably-never-get-around-2-it is to implement ssl so all tzt traffic is encrypted but i'm too lazy.

    hence that message from ur overly-dramatic browser.
     
  18. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    that iz what a phisher wud say

    and its my overly dramatic avira browser extension tyvm
     
  19. Agrul

    Agrul TZT Neckbeard Lord

    Post Count:
    43,732
    once i am happy w/ where the p2 cnn ec2 node is i may make an ami out of it

    it would be good practice to know how to do that
     
  20. Sifter

    Sifter TZT Addict

    Post Count:
    2,815
    @Chemosh would know best if he sees this post. I just put a bunch of shit in Dynamo and archive cold data to s3. Also lambda is pretty cool. That's all I got.