Chicago Fed Study Blasts the Lid Off of High Frequency Trading

We hear all sorts of reasons why the little guy won't buy stocks these days but the below graph, courtesy of a Chicago Fed study, might explain a lot of it.

hft chicago fed

The above chart shows an astounding percentage of market trades going through no human involved trading systems known as high frequency trading.

The Chicago Federal Reserve paper, How to Keep Markets Safe in the Era of High-Speed Trading, prattled off a laundry list of the most recent high frequency trading debacles including Knight Capital as well as others. Yet in spite of these increasingly frequent stock market disasters, even basic risk controls are not implemented. Why? They claim it would slow down their trading systems.

Industry and regulatory groups have articulated best practices related to risk controls, but many firms fail to implement all the recommendations or rely on other firms in the trade cycle to catch an out-of-control algorithm or erroneous
trade. In part, this is because applying risk controls before the start of a trade can slow down an order, and high-speed trading firms are often under enormous pressure to route their orders to the exchange quickly so as to capture a trade at the desired price.

While the paper focuses on events, contained within is a solid example of really bad software engineering. The Chicago Fed found code wasn't even tested, literally changes are being made on the fly on live production servers not just putting those trades at risk but the entire system as well.

Another area of concern is that some firms do not have stringent processes for the development, testing, and deployment of code used in their trading algorithms. For example, a few trading firms interviewed said they deploy new trading strategies quickly by tweaking old code and placing it into production in a matter of minutes.

Perhaps financial organizations should consider hiring some real engineers who know a thing or two about software design instead of what they are doing. No American who is worth their salt, including those specializing in advanced mathematics, would ever change, on the fly, algorithms on a live server, dealing with billions of dollars.

The study also found out of whack fictional financial mathematics, referred to as algorithms, being designed as well.

Chicago Fed staff also found that out-of-control algorithms were more common than anticipated prior to the study and that there were no clear patterns as to their cause. Two of the four clearing BDs/FCMs, two-thirds of proprietary trading firms, and every exchange interviewed had experienced one or more errant algorithms.

The report amplifies just astounding irresponsibility and engineering incompetence. Can you imagine someone in a nuclear facility implementing software changes on the fly? Can you imagine air traffic control algorithms refusing to put in safety and error checks, claiming that slows down real time air traffic routing execution?

The Chicago Fed does give some recommendations in their report:

  • Limits on the number of orders that can be sent to an exchange within a specified period of time
  • A “kill switch” that could stop trading at one or more levels
  • Intraday position limits that set the maximum position a firm can take during one day
  • Profit-and-loss limits that restrict the dollar value that can be lost.

May we add to these suggestions trading firms hire American engineers with U.S. education and training? Honestly, at least from the better universities, software development described at the level this report suggests implies a clear lack of knowledge and skill in financial mathematics and more importantly, software engineering R&D processes and procedures.

The New York Times profiled two brokers simply trying to impose a 50ms trade delay rule. They are getting attacked, seemingly by people who don't know how to develop algorithms and software:

It’s a staggering development — and one that Mr. Arnuk, 46, and Mr. Saluzzi, 45, say has contributed to the hair-raising flash crashes and computer hiccups that seem to roil the markets with alarming frequency. Many ordinary Americans have grown wary of the stock market, which they see as the playground of Google-esque algorithms, powerful banks and secretive, fast-money trading firms.

To which Mr. Arnuk and Mr. Saluzzi say: enough. At their Lilliputian brokerage firm, they are tilting at the giants of high-frequency trading and warning — loudly — of the dangers they pose. Mr. Saluzzi was the only vocal critic of H.F.T. appointed to a 24-member federal panel that is studying the topic. Posts from the blog that the two men write have been packaged into a book, “Broken Markets: How High Frequency Trading and Predatory Practices on Wall Street are Destroying Investor Confidence and Your Portfolio,” (FT Press, 2012) which was published in June. They are even getting fan mail.

But they are also making enemies.

Proponents of high-frequency trading call them embittered relics — quixotic, old-school stockbrokers without the skills to compete in sophisticated, modern markets. And, in a sense, those critics are right: they are throwbacks. Both men say they wish Wall Street could go back to a calmer, simpler time, all the way back to, say, 2004 — before the old exchange system splintered and murky private markets sprang up and computers could send the Dow into 1,000-point spasms. (The bottle of Tums Ultra 1000 and the back-pain medication on Mr. Arnuk’s desk here are a testament to their frustrations.)

They have proposed solutions that might seem simple to the uninitiated but look radical to H.F.T. insiders. For instance, the two want to require H.F.T. firms to honor the prices they offer for a stock for at least 50 milliseconds — less than a wink of an eye, but eons in high-frequency time.

Incredible that a 50ms delay is met with so much resistance, but that's how much our stock market has turned into a gambling casino based on the nanosecond response time and skew of a NAND gate in some router. Not exactly investment. Odds are if such a delay is implemented, more bad software design will proliferate, either by trying to hid trades less than 50ms and others attempting to hit the picosecond next to that time interval in execution.

Occupy Wall street and others want transactional taxes to curtail Wall Street excesses, or as they call them Robin Hood taxes. These are small taxes that occur on every trade executed, raising revenue and disincentivizing high frequency trading as well as other types of short hold trading. Needless to say a Tobin tax, or a small fee for a certain number of trades executed in a short time window, would would place speed bumps on the great high frequency trading superhighway.

One thing that can be said, Wall Street is running amok unabated. Seems the great Financial Crisis of 2008 didn't change anything. History doesn't just repeat itself these days, it repeats itself over and over again in the space of 9 nanoseconds.

Below is Bloomberg Law's exceptional questions asked interview with Senator Jack Reed (D-RI) on high frequency trading. The Senate Banking Committee is holding a hearing on high frequency trading September 20th. The SEC is holding a forum on October 2nd on the same issue.



Notice how engineers, technologists are not on the witness list and the wishy-washy hearing title, Computerized Trading: What Should the Rules of the Road Be?



This only comes years after others have known about dangers

Nice of the govt. to jump on the bandwagon years after the dangers have been known and caused repeated damage. The exchanges are now just jokes that people with inside information, access, or HFT use to their own advantage.
National governments have started looking into banning HFT because it's so dangerous to market integrity - without integrity and open, honest information, most sane rational people will avoid markets at all costs.
It's obvious the markets and banksters don't want open, fair markets. Remember the whole reasoning behind the Securities Act of 1933 and Exchange Act of 1934, protect the public through full disclosure, no lies, manipulation, or hidden BS. Again, post Great Crash close to 80 years ago, and now look how far we've gone again solely because of corruption, lobbying, and regulatory capture by the banksters/financial industry.

The NYSE, NASDAQ, FINRA, SEC, CFTC all go along with the insider game. NYSE was fined $5 million only (compare that to the money it made) for giving inside information to certain companies. That's the NYSE! Come on, when the exchange itself is breaking laws and rules, why should anyone trust some algo.?

The whole concept of running a machine that beats others in exchanges and then can crash entire stocks, commodities, or markets based on being physically closer to the exchange or some market-destroying bid and offer repeated without any intention of being executed for any normal period of time is so obviously wrong. That's about $ and access. If you can get prime real estate to exchange servers, then you "win" and beat average investors using brokers' servers? That's so wrong on so many levels. And during the "60 Minutes" show that dealt with this, the exchange acted like this was fine and the people that could get closer to their machines deserved to get their trade. How's that for fair markets?

No, zerohedge and others have been detailing this for far too long and the exchanges, regulators, law enforcement, and financial industry were screwing up markets and doing real damage to retail investors and the integrity of markets for years without any repercussions to them. Once again, certain sectors of the public are fully awake and aware and know damn well the govt., regulators, and law enforcement are late to the game and won't do anything to punish the wrongdoers or really prevent more of the same because the people that break the rules and exploit their ownership of the govt. and regulators will not tolerate any reforms.

I'm still waiting for hedge funds to be banned from handling pensions. There's no way public servants should be forced to invest in those scams. As individuals, the public servants would never be allowed to invest in hedge funds because they aren't "sophisticated" enough and don't meet the capital requirements (quite frankly, anyone who does invest in a hedge fund is a sucker). However, once they pool all of them together, then public servants can now be forced to pay out (they have no choice in who their pension fund invests with) 2 and 20 to hedge fund managers and make them instant billionaires for sitting in front of computers, following the herd of other hedge fund managers, and actually LOSING to the market indexes, even without taking into account their outrageous fees! So how's that for fairness and market integrity? In a few years when pensions go bust of course someone will shock of shocks, discover that hedge funds were probably picked because of relationships and not any real returns to the public servants. And then some politician running for office will pretend that he wants to protect the public servants and demand "real reform" in the hedge fund industry and the media will kiss his ass and call him a great reformer while he accepts $ from other banksters. Of course now those very same hedge fund managers donate to politicians that then have the audacity to berate those public servants, call them lazy, and demand they make more sacrifices while continuing to make billions for literally taking their money and losing to the indexes! USA 2012 - it's twisted.


@Kurtz - although "Lang" would be a better handle, I just wanted to say Good on ya! Super rundown. What´s to do?

More evidence markets broken - blamed on "fat finger" again

Here's more evidence the markets are completely broken - multiple oil stocks jumped 3-9% minutes before the exchange closed and then reversed gains. Still blaming the "fat finger" BS.

Funny, I thought that excuse was tried and failed a couple of years ago. Seems these excuses keep getting tried to explain away the complete manipulation and lack of credibility/oversight/honesty in today's financial markets. Oh well, if you were a small investor that bought at the top of the 9% jump, I have no doubt your trade will stand. If you are TBTF or have some other pull with the exchange, your purchases no doubt won't have to stand. Step on up, step on up, many carnival games for those willing to part with their cash.

By the way, there's an "anchor" on a certain business channel that himself had an issue with the SEC:

So, there are people like that collecting big paychecks despite the fact that they were called out by regulatory authorities. Who wants to trust the media or anyone when this is going on in front of us daily with no shame? And yet those people have very good paying jobs while they criticize Americans that simply won't get hired no matter how honest they are (or is it because they are honest they won't get hired - more and more that seems to be the case).

If you find more details on the internets

Let use know. You might format your links, just a suggestion. I'm strongly suspecting use of H-1B in this, but I'm looking for more S&T details. Fat finger, try clueless wonder.

Senate testimony

There is some interesting details on HFT from the hearing testimony. You might read it at the above link to the hearing.

I'm doing a quick scan and so far, not a word on good old fashioned terrible, negligent software design. Amazing, they are building these systems on technology, yet technologists are not present to inform them some of the events were caused by negligent R&D, forget dark pools and nanosecond algos.

don't blame the engineers

The nature of the environment does not allow for proper engineering practice. Algorithmic changes are only good till the next guy or his computer catches on. If you spend a few days testing the tiny anomaly you are trying arbitrage will be gone. The whole system is fundamentally unstable and uncontrollable. Allowing to-big-to-fail financial institutions to operate in an fundamentally chaotic environment is a recipe for disaster. The questions are when will disaster strike, how much will it cost us, and how much will the people running the show make off with before it does.

think about what you are saying

Because OMG, if one first tests their algo in a simulator, with regression tests, then they might "lose the edge" on the competition.

Think about that. There are trillions of dollars flowing through these systems that are being put at risk simply because someone needs to be "first".

It's also false, one should have, in house, a simulator, an offline testbench, not using the live, real markets, with real money that affects real people and real economies, as their testbench.

Same problem with the "bad math" CDSes, but doing that, putting new code out live, on the fly onto the markets is obviously not only playing with fire, has systemic risk.

Such practices should be banned, from a technology standpoint, for everyone.

It's rare I ever claim an engineer is negligence that this has to be one of those times, to put untested code, on the fly, out to ravage the open markets, electronic trading systems really is.

You have the wrong end of the stick

"It's rare I ever claim an engineer is negligence that this has to be one of those times, to put untested code, on the fly, out to ravage the open markets, electronic trading systems really is."

If you believe the sign-off on putting untested code on servers is down to the engineers, you are letting the real folks behind this off the hook.

It's all about management - you can bet that the culture of such practice is to not only let it happen, but dollars to donuts it will be *encouraged* to happen. The folks making the money out of this go all the way to the top, and after all, they are betting with other peoples money.

If you believe engineers can stand up to management, you are misguided. Check out Roger Boisjoly. Remember, if an engineer refuses to do something because it is stupid, he/she can be replaced. Or paid enough to hold his or her nose.

true, much higher up

Wouldn't be the first time some idiot forced some engineers to do some really unsafe stupid shit. You can tell just by the investigations, once again, the engineers are considered "the help", disposable, as if an "after thought" and that is a huge part of this problem.

That said, if a head were to roll on the engineering side, you can bet it will be the poor schmuck who wrote the code, not the people who made her/him. That said, I suspect finance thinks they can just get a bunch of foreign guest workers and it's all good, like buying a commodity, like pork bellies.

No so much as you would like to think

"That said, I suspect finance thinks they can just get a bunch of foreign guest workers and it's all good, like buying a commodity, like pork bellies."

I believe you have a foreign guest worker jag going on here. Problem here is that for HFT firms it *may* apply, but only because what they are looking for is in high demand and short supply.

Put another way, I friend of mine works for a HFT firm in NYC. They pay 2x-3x the prevailing salary for a programmer with the right skills (Linux kernel and assembler language knowledge). They don't pay that kind of salary because they are 'like pork bellies' - and if they have to go overseas to get that they will.

Would they throw that engineer to the wolves for a screw up? Well, given the income of those higher up, you bet. The point is as always: follow the money.

makes it even more pathetic then

Clearly they didn't "get that". Sorry, even if someone stumbles over for loops, they would know not to test their code in a live production environment.

asm + kernel, I sincerely doubt there is a shortage since anyone with a BS CE, soft focus, from a top school should have those base skills + plus the base statistics and probability theory.

Sure engineers are viewed as pork bellies, happens every day even with $100k+ base. Just look at the sad state of hiring with keyword rejection, bogus "gotcha" tests that people get the answers to beforehand. Here every day how these things reject the solid engineers and pass the bad.

Bottom line, this is no joke, you're talking about bad methodology and practice where trillions of dollars are vulnerable. This isn't a kernel panic, it's trillions of dollars.

This is a situation where every day you have a flash crash doing on. Bad code and bad algos are not a rarity, not a fluke here.

If they are using kernel mode and asm makes it even more obscene to do on the fly code changes. You don't test assembly on the fly, live or even write assembly past 20 lines w/o running a unit test. I'll bet money at this point these NFT firms did not take the time to build a simulator.

Engineering financial systems is the same as designing an airplane. If a company does short cuts and thousands of people fell out of the sky to their deaths as a result, you can bet we'd have laws and lawsuits for safety, regulations.

Yet when describing market disaster, we're seeing excuses here for what keeps being brought to light. It's simply very poor engineering practices.

You obviously didn't read my reference to Roger Boisjoly

You may sincerely doubt that there is a shortage of these folks yet they are paying 300k a year for those skills. Ipso facto, they are not plentiful - if they were, they would be paying prevailing wage of 100k.

Be that as it may, I am not disagreeing with you about what happens and that it is bad. What I disagree with you on is that the engineers are in control of it. They are not - it is management. Pure and simple.

But it is always easy to blame the folks at the coal face, versus those really running the show. It is symptomatic of what is wrong - the entry level folks get put to the sword, while those in charge dodge the bullet and take the payout. Remember, this is not about engineering practices, this is about making money. You can see it in other industries too - big pharma, big agriculture, big oil. It goes on and on.

As for engineering an airplane, check out the reference to Roger Boisjoly that I mentioned before. So much for engineers in charge, right?

my point, pork bellies

If engineers are not in charge, usually through a CTO at the top,VP of Engineering, then team managers, team leaders, that is what I mean by treating engineers as pork bellies, just another commodity. It doesn't matter if they are paying $300k if they have no power over R&D, then it's usually a disaster.

One of these fellows who is

One of these fellows who is actually in the business suggests to me that the prevailing wage is not an artifact of shortages or labor pool elitism (although they certainly do target the already-wealthy), but the fact that *they know where the bodies are buried*. Implicit in this high salary is never talking to the press, never leaving the company hanging as the sole proprietor of a piece of code, and never working for an opposing company.

non-compete clauses

I don't know about NYC, CT, NJ and so on but non-compete are not enforceable in CA. That said, most engineers have NDAs, confidentiality agreements, clauses to even come and support patents after leaving the firm. Hanging as sole prop. on code as a work product sounds strange, as if there is a liability clause hanging the disaster on that poor, unprotected, since not incorporated, schmuck?

Serially, there are trade secrets, proprietary algorithms and so on and most engineers worth a damn know absolutely not to talk or to leave src unsecured, anywhere near an live port...or walk into a bar and leave their beta prototype (:)) and if they have an ounce of ethics, uh, not download it and dump it onto a flash drive the last day at work. ;) Yet, putting untested algos and code on a live production server/router is insane, so how could these firms do worse, unless they forced that as which point they should be liable?

To me this is just astounding considering these reports of such shoddy engineering, software/algorithm work. It's kind of amazing people are performing this poorly on an engineering level and pulling $300k, sorry but even for mega rich Manhattan that just seems bizarre.

Goldman Sachs criminal charges against "programmer"

God, it irks me to see software engineers called "programmers". If this guy is so brilliant, how come he was so stupid to ftp proprietary code? I would assume all know every server captures every single event in logs and is traceable? Pretty damn stupid for someone who was so "valuable".

More proof law enforcement serves banksters, not the public

It's pretty interesting the Manhattan DA and feds seem to be doing the bidding of Goldman Sachs and other banksters. Again, did the Manhattan DA or other law enforcement authorities bring any charges whatsoever against the banks implicated in money laundering for the drug cartels and terrorists (e.g., Standard Chartered, HSBC, Wells Fargo, etc.) that all have HQ or branches in Manhattan? Even after these banks settled civil actions, there are no criminal prosecutions, espcially against key officers. Those crimes affect US citizens directly and violate US federal law and NY State law and the public in general. It's absolutely disturbing that the NY DA is focused on one person merely because he dared to go against Goldman Sachs. Apparently helping terrorists and others isn't a concern for the DA located in the place attacked on 9/11, but protecting banksters' IP is? The facts and prosecution really do speak volumes.

twice no less

The Federal Conviction was overturned so now the State is having a do-over. Excellent point, prosecutors wasted no time going after this guy whereas MFGlobal, where it's pretty clear we have brazen customer theft, not only is Jon Corzine walking away Scot-free, I read somewhere they were going to do a new hedge fund.


Supposedly this guy claimed he was downloading "open source" and since it appears these trading algorithms are modifying server software, which is extremely "open source", i.e. CentOS (I hope, most stable), that's the claim. Now it depends on the GPL license of the code, which may make a defense. In some cases modifications are "sticky" and are covered under the GPL umbrella. I don't think so here, I think this guy simply walked out with the code, thinking there wasn't a problem to take one's work "with you" to a startup.

But that said, yes, we sure do have different rules going on here for engineers vs. hedge funds and Wall Street. Theft of them is prosecuted to the ends of the earth, theft of us is just the way of doing business.

Oh, it's obviously out of control regarding differing treatment

If I was this guy's attorney, I'd just invoke Jon Corzine, Dimon, and everyone else. Shoot, just for kicks, I'd yell "Occupy Wall Street" for fun on the courthouse steps and during pretrial hearings. Now that would get anyone held in contempt and wouldn't really be relevant to anything going on regarding the case, but screw it, I think most sane people hold the whole show in contempt now, Lord knows the big boys like Corzine hold us in contempt. And maybe saying it enough would mess with the jury and bring on some nullification (not sure it would work for a former GS worker, but it might for a regular citizen screwed over by GS). I hope the guy's attorney does find out who and what lobbied so hard to get the charges brought one and two times. Opensecrets does show people are able to spend big and directly lobby the DOJ/US Attorneys' offices, and no doubt Cyrus Vance's door at the DA's Office is open to campaign contributors too. Now that might not help low-level criminals, but in bankster Mecca NYC, it surely must get his attention. The same way Goldman can push for criminal prosecutions is the same way Corzine and Dimon can prevent prosecutions. Honest career prosecutors (those not seeking big law jobs defending the same banksters after they serve the public) must have some unease about prosecuting these cases knowing full well people that launder terrorist money or steal billions from farmers are walking free while the DA purposely ignores their crimes. Wouldn't be surprised if the prosecution handling this case seeks work at the banks after their time is up. At least we should hope there are enough Assistant DAs that see the charade and are fighting against it in the DA's office.

Is stealing information that bones clients really "valuable"?

So the guy downloaded code, so what? Goldman Sachs has been proven time and time again to bone clients, bet against them, sell them on products they are betting will fail, etc. So this guy took, stole, whatever (apparently the Feds think it's not theft, and given the supremacy of the feds, NYDA should listen up lest they waste 5+ years fighting this through the Appellate Div., then NY Court of Appeals, then 2 CCA), so what? If it's as worthless as everything else Goldman Sucks produces, it's actually a detriment to anyone this guy pawned it off on, even if they were willing to pay him six or seven figures. Cyrus, offer the guy discon under 240.20 or an ACD with a promise not to break the law. Hell, if Goldman Sucks can fix prices in commodities or bet billions against private and public clients, surely the NYDA can cut this guy a break. And to all ADAs in the NYDA's Office - remember, do justice, it's the job, not serving the money - if you can't do that, quit. Do you share more in common with the Americans on the street struggling against banksters and crony capitalism or with the man who probably got his job because he's the son of a former Secretary of State and sure as Hell has connections most people could never dream of? And Cyrus and all other law enforcement across the country, we're waiting for the money laundering prosecutions, conspiracy to aid terrorists, conspiracy to aid drug traffickers, mail and wire fraud, price fixing of commodities, fraud, commingling of funds, grand larceny (hundreds or thousands of counts), forgery, etc. We are awake and aware and have the skills and knowledge. If you won't prosecute these folks, then by all means help prosecute Pussy Riot, they might be released in 2 years.

There is huge interest in this article

We're seeing this article be retweeted extensively. What the question really seems to be from reading the retweets is what exactly are the engineering practices of HFT platforms and the "safety" thereof?

We reviewed what is verified, via the Chicago Fed as well as testimony. That said, since clearly the engineering details, from software, to architecture to actual trading algorithms, triggers are not really being examined in terms of systemic risk as we noted in this article, we're happy to look at more details including professional publications, patents and even architecture design docs. We happen to have in our bag of tricks some technical chops, so we'll be happy to overview more technical facts as we find out about 'em. They just need to be verified, we can't do much with "my friend said" or "supposedly these guys are hacking up CentOS in their garage" or all of the various claims I've seen flying in the comments and out there on the Internets.

Transaction Tax

Could this be the answer?

Its pretty simple and would cut down on millisecond computer driven speculation.

"Ellison’s bill would raise up to $350 billion through a small tax on stock, bond, derivative and currency trading."

Nobody would suffer with this sane way to raise revenue.

transaction tax is a tobin tax

We list those above and if you think Wall Street has gone ballistic over even increased capital requirements, OMG a transaction tax or small fee once a trade has hit a certain number of times in x predefined time window will cause them to go absolutely ballistic. We're all for a transaction tax of course, cleverly crafted but the army of lobbyists Wall Street will unleash should look like the "Red army" swarming the hill.

Needless to say, Dem leadership has shot this down and of course the Obama administration will. Can insult the donor class. I think it's a very good idea and a tool that could be utilized to help stop systemic risk as well, but I don't think it has a prayer's chance of passing since our government is owned by Wall Street.

Europe votes for 1/2 second delay on HFT

European lawmakers just voted to enforce a 1/2 second delay on trades. This is 10x the time length of what is being proposed in the U.S. (and fought, ignored "studied").