Discussion in 'Tech Heads' started by Agrul, Nov 13, 2017.
any1 got any
that's not too broad at all.
next post from agrul:
"computers = any tipskthx"
it's intentionally broad u shitty IT person
do u have tips for saving money on storage / compute?
tips for saving money during dev before moving to a pricier node/cluster for real work?
great articles on the differences between EBS and nodes that come with standard SSD
observations on what aws nodes cannot do? (eg recvently discovered they doesnt seem to have any kind of graphical terminal available, 2 the point that it is apparently meaningless to install python's tkinter on them)
do u have fancy spot pricing rules? are ur spot pricing rules the next chain linkz?
what do u use s3 / ec2 / emr for? anything interesting ppl wouldnt normally think of?
gimme ur secrets all of them
this was prompted by my recently:
A) spinning up a p2.xlarge gpu ec2 node, doing some work on it
waiting 2 days
B) finding my limit for p2.xlarge nodes had suddenly dropped to 0
D) talk 2 aws ppl
E) get non-0 limit
F) no idea why i was able to run one in A)
anyway this made me think there are probably many quirks to aws i am unaware of so talk about all of them
there are so many quirks it would be hard to know where 2 start. i'll try to blast some answers 2 ur questions shortly though.
also, i've not used a p2.xl gpu node but i'm going to guess it's going to cost about 1 arm + 1 leg 2 run per/hr
big agrul w/the big budget
big agrul = government bloat
use windows desktops, not aws - make america great again
p2.xlarge isn't too expensive, still <$1 / hr typically, and with spot pricing i have gotten them for like $0.2-0.4 / hr mostly
i mean it would be expensive to run one for a month straight 4 sure, but that's not my plan u know
but yah the cost is why the 0 limit apparently
to protect us from ourselves + scammers, altho still doesnt explain why i was able to create one originally
lookin at my graph
it would appear in fact that i got 1-2 hrs of p2.xlarge for either $0.17 or $0.34
that seems p reasonable to me
dat $0.47 is way wrong tho imma make p2 do some wurk this weekend
not rly, except to have automation to shut stuff down if it's not being used. that's a rly obvious tip that in practice is an enormous amount of work for tech shops that are not used to automation.
also, watch ur bills like a hawk. i can't think of any company i have worked for that did not get sticker-shock at least once because they just didn't realize how costs would skyrocket until bezos already owned their souls.
again nothing specific, just the obvious - for functional dev don't use full size instances if u don't need it.
honestly, AWS docs are generally excellent and it would be hard to do better than just reading what amazon already publishes. that said, i'm not sure if ur differentiating between EBS and ephemeral SSD. basically ephemeral anything just needs to be considered during achitecture stage. probably one of the most common newbie mistakes ppl make is writing stuff to some ephemeral local disk then crying after they reboot and lose all their shit.
i've heard that nesting virtualization layers on EC2 can be problematic/impossible depending on what ur trying to do. i can't imagine you'd be doing that though so not relevant probably. i can't rly think of much else outside your own discovery.
well, actually, i'll say that observing network traffic and troubleshooting network oddities is a place where AWS is a pain, because in-house you can get switch/router logs and counters and stuff and in AWS none of that stuff is really exposed, so it can be a bit hairy if you suffer weird/random network probs.
another thing that's tough at scale (network-related) is knowing what public IPs ur AWS stuff is going to use. this is irrelevant for most ppl, but in cases where you always need to know your outbound IP from AWS (for example if u need to IP whitelist that traffic), those ips change all the time. you can work around this using NAT gateways and assigned elastic IPs, but it's best to do this up-front from the beginning if u know ur gonna need it. This is just for ec2 shit. I don't even know if it's doable w/the other 2340932409203234 services AWS offers.
i personally have not dabbled in this, but yes ppl do make a science out of spot pricing shenanigans. i've worked at a place that had a whole department soley dedicated to gaming AWS spot pricing to save money, and i'm sure despite the cost of those engineers it was probably well worth the investment.
ec2 runs TZT. been p good so far.
s3 has been used for all the things in my past 4 workplaces. too many use cases to describe. make sure 2 secure ur buckets or else you'll get pwned.
emr seems useful but for some reason we decided to go w/databricks @ work... that's been pretty solid (because i think a lot of the apache haderp/spark original developers were the ones who founded databricks)
my understanding is that databricks does a better job optimizing spark jobs than native spark in emr (the databricks ppl claim rather dramatic speedups on some jobs, something like an order of magnitude or even two iirc); i think i would rather be working in it. emr has the advantage that it is lazier to setup though (very important in comically bureaucratic work environments)
does databricks have as nice a range of machine images available? the rather complex software (e.g. tensorflow and theano and torch and ... u know all the gay machine learning software; very strangely tho they dont have good AMIs for operations research or standard stats etc) available pre-installed on ec2 AMIs is pretty impressive. the reason i started dicking around with this p2.xlarge for example is because i'm giving a talk at the end of this month about convolutional neural nets, and i wanted access to a gpu to do that, but i didn't want to try to get theano or any of the other big data frameworks functioning on my home pc (for a # of reasons, mainly: currently wnidows pc & 2 lazy to setup another dual boot right now but windows sux at real good softwares, 2ndarily bc i have a nice-ass gpu but it is driving my nice-ass monitor and running compute off the same gpu driving ur monitor is itself a world of headaches, especially in windows)
ec2 runs tzt? you have like an ec2 node running that acts as tzt's server? what are the monthly costs like on that?
do you have a preferred way of spinning up short-lived nodes/storage and moving homegrown software to them? for the cnn talk i'm giving i've settled on moving shit to a git repo (including data, since i picked a small example that's still shy of 1GB, though not by much) and cloning to the ec2 node whenever i spin it up. previously i wrote a python script to do essentially the same thing but the git workflow seems lazier and more fuckup-proof
how come u phishin me utumno
i remember a p wide range but am too tired/lazy 2 look up now. may respond to this later in the week when i'm looking at our shizzy and already logged in
yes. ec2 node running xenforo - a t2 medium, which is admittedly overkill for our needs but i wanted 2 be able to add stuff if needed + being new to xenforo i didn't know if it was as lightweight as promised. also using ebs storage (80gb gp2 volume), which seems plenty - 240 iops which can burst to 3k if needed. this is shitloads better than the old host where we would frequently max out iops and the mysql db would slow 2 a crawl. price? i also have a t2.nano bastion host so i'm not 100% sure but i'm guessing it's ~$30/mo or so just for tzt server.
i do not. i was lazy and did all the setup by clicking buttons and shit in aws console, and i told myself i would either use cloudformation or terraform + ansible to automate all the things, but never got around to it. i hear cloudformation is probably the ezmode way of doing automated spinups then use ansible 4 config-mgmt on ur new nodes. i guess even easier/lazier is to just save your Known Good Node as an AMI (if we're strictly speaking EC2 only here) and then spin those up/down as needed and not even bother w/config mgmt or host automation beyond start/stop scripts - but that's obviously more inflexible since every time u need to make changes you'd need to generate new ami and fuck that.
on my to-do-list-but-will-probably-never-get-around-2-it is to implement ssl so all tzt traffic is encrypted but i'm too lazy.
hence that message from ur overly-dramatic browser.
that iz what a phisher wud say
and its my overly dramatic avira browser extension tyvm
once i am happy w/ where the p2 cnn ec2 node is i may make an ami out of it
it would be good practice to know how to do that
@Chemosh would know best if he sees this post. I just put a bunch of shit in Dynamo and archive cold data to s3. Also lambda is pretty cool. That's all I got.
Separate names with a comma.