You are currently browsing the tag archive for the ‘Algorithms’ tag.
Facebook, CFTC – U.S. Commodity Futures Trading Commission, here. First there is a facebook page for the CFTC, that’s one good thing. Go click Like on that now.
Lipton, GLL, Predictions and Principles, here. 2013 Predictions. That’s two good things.
Matt Levine, DealBreaker, New Reporting Requirements Let You Watch Swaps Trading In Fake-Real-Time, If That Sounds Like Something You’d Like, here. That’s, Three! Three good things.
What should we do with the new real-time swaps data depository? As of the new year, the CFTC is requiring all swaps dealers to report their swaps trades in real time, starting with rates and credit index trades, and here is the data depository with those (anonymized) reports. It’s … not particularly real-time! It’s not particularly user-friendly either, I gotta say; somebody is presumably planning to make a lot of money translating these zip files of haphazardly coded data into Useful-ese.
DTCC, Real-Time Dissemination Dashboard, here. Rates .zip files but the data might be kinda crusty.
Don’t, like, believe these numbers or anything. The “cumulative slice” seems to be all trades made yesterday, plus some filler. I attempted to slice to just USD/USD trades but the data is full of weird coding so you can never be quite sure; there’s a nontrivial amount of “Fixed/Fixed” swaps here which seems suspicious. Also these are not just swaps; they include forward rate agreements and also various sorts of options and exotics that may or may not be usefully captured in the depository. (Matt Levine)
The OTC Space, Watch the OTC Market in Real-Time, here.
For the first time in recorded history you can now watch OTC trades occur in “real time” via the DTCC Swap Data Repository, based in the US. The website provides anonymised data snapshots of market activity for Credit and Rates trades, plus a “real-time” ticker showing trade events being reported as you watch. I say “real time” as there is a considerable amount of technology and processing going on within each reporting firm, and within DTCC, before the trade events pop onto the screen. The portal can be reached here:
DTCC, DTCC SWAP DATA REPOSITORY REAL-TIME REPORTING NOW LIVE, here. Lots of shouting, Let’s wonder together, why?
New York, NY – January 3, 2013 – The Depository Trust & Clearing Corporation (DTCC) announced today all registered swap dealers active in credit and interest rate trading are now sending information to DTCC’s swap data repository (SDR), DTCC Data Repository (DDR). On December 31, DDR began accepting data from swap dealers for over-the-counter (OTC) trades as outlined by the Dodd Frank Act (DFA) and the Commodity Futures Trading Commission’s (CFTC) real-time and regulatory reporting rules.
DDR is now publishing real-time price information. Since the December 31 swap dealer reporting deadline, DDR has disseminated more than 10,000 records, which represent the vast majority of the reportable OTC derivatives market. Reports are available through file transfers, RSS feeds and internet access to a ticker page, Excel and search functions on DRR’s website,
Colfax Research, Arithmetics on Intel’s Sandy Bridge and Westmere CPUs: not all FLOPs are created equal, here. Might be Treasure. Starts to quantify what AVX brings to the table in terms of optimized performance.
This paper presents a new arithmetic efficiency benchmark and uses it to compare the Intel Sandy Bridge E5-2680 CPU to the Intel Westmere X5690 CPU performance. The efficiency is measured for single and double precision floating point operations: addition, multiplication, division, square root and the exponential function, and for 32- and 64-bit integer operations: addition, multiplication and division. The SSE2 and AVX instruction sets, as well as scalar operations, in single-threaded and multi-threaded modes are covered. This benchmark eliminates the effects of memory bandwidth and latency by fitting the calculation in the L1 cache. The bandwidth of the L1 cache and main memory (RAM) are estimated for reference, and the LINPACK benchmark result is reported.
TEDx NewWallStreet, one message “Banking is ripe for Information Age innovation, disruption and improvement.” , here. Oh, ripe eh? TED talk videos on Finance.
Blodget, Business Insider, EXCLUSIVE: Here’s The Inside Story Of What Happened On The Facebook IPO, here.
In one of the biggest IPOs in history, in which a huge amount of stock was sold to small investors, privileged Wall Street insiders once again got top-notch information…and individuals got the shaft.
Zerohedge, An $8bn Loss Or Was JPMorgan ‘Unhedged, Long-And-Wrong’ Post-LTRO2? here.
So, in summary, it appears that the CDS data confirms what we suspected.
- A large (~$120bn) tail-risk tranche credit hedge was placed.
- The hedging of that hedge became very onerous but surprisingly profitable as markets rallied day after day with no give-back.
- This led to a greedy trader lifting some of the original tranche (and the HY short side) and leaving himself much more naked long to the market into LTRO2 – which marked the top. Losses escalated through April (~$2.5bn or so).
- Dimon went public (with some of the details).
- Last week, the rest of the tranche was dumped (we suspect) at a large cost (perhaps ~$5.5bn) leaving, we suspect…
- A potential ~$8bn loss and a heavy IG9 long credit position hedged (with major basis risk – difference in dynamics between the legs of the trade and the hedge) by various other liquid positions including shorts in HYG, JNK, IG18, and HY18 (and we would suspect equity/financials too).
Morgan Stanley’s research team came out with a note on Friday, guesstimating that JPMorgan’s losses on the synthetic credit portfolio held by its Chief Investment Office will come to $5bn by the end of the year, which is $2bn more than CEO Jamie Dimon seemed to think they’d come to when the announcement of the losses was first made on May 10.
I might have enjoyed this Andrew Ross Sorkin column, about how bringing back Glass-Steagall would have prevented neither the financial crisis nor l’affaire Whaledemort, more than most people.* Yes, the argument is pretty silly – like saying we shouldn’t have speed limits because they probably wouldn’t have prevented the Columbine massacre – but it contains an essential point that can’t be made often enough about the Volcker Rule:
But [bad sh]it often starts with banks making basic loans. Making loans “is one of the riskiest businesses banks engage in and has been a major contributing factor to most financial crises in the world over the last 50 years,” Richard Spillenkothen, former director of the division of banking supervision and regulation at the Federal Reserve, wrote in a letter to Politico’s Morning Money on Monday.
Aaronson, I was Wrong about Joy Christian, here. Sometimes the snark has to go to eleven.
In response to my post criticizing his “disproof” of Bell’s Theorem, Joy Christian taunted me that “all I knew was words.” By this, he meant that my criticisms were entirely based on circumstantial evidence, for example that (1) Joy clearly didn’t understand what the word “theorem” even meant, (2) every other sentence he uttered contained howling misconceptions, (3) his papers were written in an obscure, “crackpot” way, and (4) several people had written very clear papers pointing out mathematical errors in his work, to which Joy had responded only with bluster. But I hadn’t actually studied Joy’s “work” at a technical level. Well, yesterday I finally did, and I confess that I was astonished by what I found. Before, I’d actually given Joy some tiny benefit of the doubt—possibly misled by the length and semi-respectful tone of the papers refuting his claims. I had assumed that Joy’s errors, though ultimately trivial (how could they not be, when he’s claiming to contradict such a well-understood fact provable with a few lines of arithmetic?), would nevertheless be artfully concealed, and would require some expertise in geometric algebra to spot. I’d also assumed that of course Joy would have some well-defined hidden-variable model that reproduced the quantum-mechanical predictions for the Bell/CHSH experiment (how could he not?), and that the “only” problem would be that, due to cleverly-hidden mistakes, his model would be subtly nonlocal.
What I actually found was a thousand times worse: closer to the stuff freshmen scrawl on an exam when they have no clue what they’re talking about but are hoping for a few pity points. It’s so bad that I don’t understand how even Joy’s fellow crackpots haven’t laughed this off the stage.
Coursera, website, here. Jeff Ullman is teaching Automata; It’s like Gandalf is coming back to the Shire to teach the Hobbitses. Sry, cannot do this all in LOTR metaphor, perhaps we need to shift metaphor to nail down the dynamics here. These Coursera guys are like the Lakers. Knuth is letting Sedgewick be the Algorithms guy, very Kareem/Bynum. Need Bill Simmons to pick this up at Grantland to do this right.
edX, website, here. These edX guys are the Celtics. Agarwal is fronting this now and he is a Rondoesque stud, but where are the big three? Wait until they finish negotiating with Michael Sandel and Justice maybe we see the big three then. Sandel is like the LBJ of online education (certainly more than a Paul Pierce equivalent). Sandel should be inline for a Lebron James-style payout here. edX better be offering a MAX+ contract there as well as the Harvard economic equivalent of a shoe deal, whatever that might be. Maybe he gets his Chair multiply endowed, or something. Is it recoverable if he decides the right thing to do is not sign with edX? Does Sandel have a posse? How great would that be if Sandel had a press conference to announce “I’m taking my talents to South Beach.” The beantown fans would revolt and burn their Justice jerseys and backpacks; they’ll have to start an edX Cares set of television spots where Mankiw and Stallman talk about how personally gratifying it is to give back to the community.
Udacity, blog, here. The Heat? This could be their year or the only defender that works against these guys is the fourth quarter. These guys better do a deal with Oxford/Cambridge kind of soon. Looks like UCB signed with Coursera.
An interactive online learning system created by two Stanford computer scientists plans to announce Wednesday that it has secured $16 million in venture capital and partnerships with four major universities.
The scientists, Andrew Ng and Daphne Koller, taught free Web-based courses through Stanford last year that reached more than 100,000 students. Now they have formed a company, Coursera, as a Web portal to distribute a broad array of interactive courses in the humanities, social sciences, physical sciences and engineering.
Besides Stanford, the university partners include the University of Michigan, the University of Pennsylvania and Princeton.
NYT, Harvard and M.I.T. Team Up to Offer Free Online Courses, here.
In what is shaping up as an academic Battle of the Titans — one that offers vast new learning opportunities for students around the world — Harvard and the Massachusetts Institute of Technology on Wednesday announced a new nonprofit partnership, known as edX, to offer free online courses from both universities.
Algorithms, Sedgewick and Wayne, 4th Edition, here. They’re covering the online free course trend starting in Aug 2012, here . They are going to sell some books, I think. First, they have the Princeton Logo for the course, that’s a big deal I suspect. Second, isn’t this a case where the dog and the tail are mixed up? We’re leading with the Book (the tail) and then mention there is this course thing (The Dog) you are probably not interested in. So where do we mention that course, the one that Bezos took before he went to Amazon? Somewhere toward the end of the announcement where people won’t see it. And lets make it two courses so they have to sign up for each one, in case they don’t like the first one.
The Headline announcement should be:
Psst, Kid wanna do Algorithms with Sedgewick? Wouldja look at this? Here is a free internet course starting in August, direct from Pee – rince – ton Univer- si – tay. U in? …good!
Oh, and there’s a book that will help you solve the homework problems at Amazon, buy it if you need it. Gotta go kid, see ya.
Irving Wladawsky-Berger, Blog, here. Head of IBM Research back in the day. Probably worth tracking to see where it goes.
Technology Review, Moore’s Law Lives Another Day, here. Confirms what was bugging me, the 3D lithography techniques have been around since the late 80s in Japan. Used to see it in DRAM manufacturing presentations. 22 years to volume production in 22nm process for x86. Wired, April 19, 1965: How Do You Like It? Moore, Moore, Moore, here. Puff piece, but it has a link to Moore’s 1965 paper in Electronics magazine.
Next-generation 20 nm processes can support optimized versions for low power and high performance, according to an IBM expert. GlobalFoundries will decide in August whether or not it will offer such variations.Those were just two data points from wide ranging discussions at the GSA Silicon Summit here. Separately, executives said a variety of 3-D ICs will hit the market in 2014 despite numerous challenges, and CMOS scaling is slowing down but still viable through a 7 nm node.“Recently TSMC said at 20 nm there are no significant differences [in process optimizations], but I don’t believe that,” said Subramanian Iyer, an IBM fellow and chief technologist in its microelectronics division. “I believe at same node you can have two [different variations],” he said in a keynote here. Indeed, GlobalFoundries is debating whether it wants to offer high performance and low power variants of a 20 nm process it is putting in place today.
Fabless FPGA vendor Achronix Semiconductor Corp. (Santa Clara, Calif.) has announced details of its Speedster22i HD and HP product families, claimed to be the first FPGAs to be built on a 22-nm manufacturing process technology.The devices are the result of a foundry agreement with Intel Corp. announced in November 2010 and the first devices are due to sample in the third quarter of 2012.Both the HD (high density) and HP (high performance) families come loaded with a variety of high-speed data communications interfaces hardwired. These include 10/40/100G Ethernet MACs, 100Gbit Interlaken channels, PCI Express and DDR3 memory channels that run at up to 2133 Mbps. In the case of the HD1000 device these are two, two, two and six respectively. This optimizes the Speedster22i FPGAs for work in networking and telecommunications equipment although the company stresses that the devices can find applications in servers, high-performance computing, military, industrial and scientific applications. The large number of high speed memory channels provides the industry’s highest bandwidth FPGAs, Achronix claimed.Achronix’ existing product range is based on 65-nm process technology and the move to Intel’s 22-nm FinFET process allows Speedster22i family to consume half the power at half the cost of high-end, 28-nm FPGAs.
Wired, How to Spot the Future, here.
This may sound like a paradox. Surely technology always promises something radically new, wholly unexpected, and unlike anything anybody has seen before. But in fact even when a product or service breaks new ground, it’s usually following a familiar trajectory. After all, the factors governing thermodynamics, economics, and human interaction don’t change that much. And they provide an intellectual platform that has allowed technology to succeed on a massive scale, to organize, to accelerate, to connect.
So how do we spot the future—and how might you? The seven rules that follow are not a bad place to start. They are the principles that underlie many of our contemporary innovations. Odds are that any story in our pages, any idea we deem potentially transformative, any trend we think has legs, draws on one or more of these core principles. They have played a major part in creating the world we see today. And they’ll be the forces behind the world we’ll be living in tomorrow.
Noahpinion, Thursday Roundup, here, I need to know more about how money market funds and commercial paper broke in the credit crisis – he points to Cochrane on money markets which is a start. The idea is to sort out where bank runs can happen as confidence evaporates. What breaks first next time? Noah Smith could be the EcoFin summary guy after DeLong.
HPC Wire, Some Thoughts on Intel’s Acquisition of Cray’s Interconnect Technology, here. They lead with:
The reasons for this deal, in my opinion, are as follows:
The general trend to commodity components continues. For small companies like Cray (with about 800 employees), it is simply too expensive to innovate and develop sophisticated hardware such as an interconnect for exascale computing. And Intel is certainly able to take this on, especially now with all the expertise gained from the previous QLogic acquisition and now from the 74 interconnect experts moving from Cray to Intel.
Craig’s List, I will legally change my name to yours for a WWDC ticket, here. I like how Gruber posts stuff that induces Karl Denninger (here) to call a market top on AAPL, Gruber records it in Claim Chowder on Daring Fireball, and then Gruber spikes the unfortunate Karl Denninger 6 months later. It’s like who killed Kenny in South Park. I am worried however that John Gruber is just an alias for Karl Denninger, which would make the world a smaller, less predictable, and meaner place, so I won’t think about that.
Turing’s Invisible Hand, I grade grad AI, here. Nice slides from the course.
This semester I have been co-teaching (with the awesome Martial Hebert) CMU’sgraduate artificial intelligence (grad AI) course. It’s been a lot of fun teaching AI to a class where a significant fraction of the students build robots for a living (possibly some of the students are robots, especially the ones who got a perfect score on the midterm exam). Although the last class is on May 2, I already gave my last lecture, so this seems like a good time to share some thoughts.
My general impression is that many AI courses try to cover all of AI, broadly defined. Granted, you get Renaissance students who can write “hello world” in Prolog while reciting the advantages and disadvantages of iterative deepening depth-first search. On the down side, breadth comes at the expense of depth, and the students inevitably get the very wrong impression that AI is a shallow field. Another issue is that AI is so broad that some if its subdisciplines are only loosely related, and in particular someone specializing in, say, algorithmic economics, may not be passionate about teaching, say, logic (to give a completely hypothetical example).
Business Insider, BLANKFEIN: The Only Reason Goldman Got Into Trouble Is Because Our Competitors Sucked At Risk Management, here.
DealBreaker, Marvel At The Derivative On Its Derivatives That Credit Suisse Wrote To Itself, here. This looks like the mezz tranche CS awarded for end of year compensation a couple years back. I stopped reading Deal Breaker for a while, but Levine has been very solid recently.
Business Week, Stock Trading Is About to Get 5.2 Milliseconds Faster, here. 59.6 milliseconds NYC to Lon roundtrip latency.
HPC Wire, Intel Makes a Deal for Cray’s Interconnect Technology, here. So Cray wants out of the interconnect hardware business.
Supercomputer maker Cray is methodically and inevitably shifting its technology focus from hardware to software. Another step in that direction played itself out this week in the company’s sale of its highly treasured supercomputing interconnect technology. On Tuesday evening, Cray and Intel announced that they signed a “definitive agreement” that would transfer the interconnect program and expertise to the x86 chipmaker.
Cluster Monkey, Cluster Interconnects, here.
This article will focus on interconnects that aren’t tied to vendor specific node hardware, but can work in a variety of cluster nodes. While determining which interconnect to use is beyond the scope of this article, I can present what is available and make some basic comparisons. I’ll present information that I have obtained from the vendors websites, from information people have posted to the beowulf mailing list, the vendors, and various other places. I won’t make any judgments or conclusions about the various options because, simply, I can’t. The choice of an interconnect depends on your situation and there is no universal solution. I also intend to stay “vendor neutral” but will make observations where appropriate. Finally, I have created a table that presents various performance aspects of the interconnects. There is also a table with list prices for 8 nodes, 24 nodes, and 128 nodes to give you an idea of costs.
Slashdot, Travelling Salesman, Thriller Set in a World Where P=NP, here. Watch the trailer – it is the best thing you will see this week. Premieres 16 Jun. From the trailer it looks like Tom Cruise meets Mr. T to work a polynomial runtime algorithm to take over the world. Wonder what language the algorithm is coded up in? Please, please have it be standard ml of new jersey running on Xcode/Mountain Lion cross compiled to run on Amazon Elastic Compute Cloud. If these really are literally four of the smartest guys on earth you know right away there is going to be a problem checking out the right patch version from CVS so they know what to build and hide from the bad government drill-your-head guys. Maybe there are a bunch of grad students in the next room, with pizza and iPad2s or something. I’ll start you off with the relevant dialog:
Government Bad Guy: Hit Control-C now.
Smartest in the World 1: I can’t …
President: NOW! Abort!
Smartest in the World 2: What’s Control See?
A small informal effort like Pink Iguana needs to lean heavily on curation for a specific audience. How narrow is the audience? Take the entire massive EcoFin community of DeLong, Wilmott, and Mankiw then subtract most of the folks who: don’t care if a 22nm semiconductor fab is competitive in 2012, haven’t compiled their code –O3 recently, or are sort of meh to the idea that there is a RDMA transport to L3. Those folks remaining might be the Pink Iguana audience if they also like: buying credit protection from AIG stories, P=NP speculation, and IEEE754. It’s the far side of the long tail.
So why curate for such a specific audience? Despite The End of Blogging, in 2012 there are remarkable and reasonably frequent publication streams from Gowers, Lipton, and Tao. The thing that is different in the last five years is the public availability of unfiltered, authoritative, and lucid commentary on specific topics. The keys are unfiltered, authoritative, and lucid. DeLong, Mankiw, Cowen, and Krugman run similarly authoritative and lucid publication streams that are more informed by their partisan backgrounds than Gowers, Lipton, and Tao. Intel, NVIDIA, and IBM have authoritative and lucid information as well, but they also have a day job to do. If folks like Gowers, Lipton, and Tao are regularly publishing there might be more, right? You just have to go look around, and maybe you figure out how something (e.g., ETP Arbitrage, Credit Derivatives, HFT, a specific floating point computation) actually works. So, Wisty curates on Pink Iguana.
Why are these folks in the Pink Iguana Hall of Heroes (listed below the Blogroll) and why should you read the Heroes?
A Credit Trader hasn’t published since 2009, he went to do other stuff, but wow what got published there was magnificent. Read Getchen Morgenson at NYT, for example this, then read The AIG Fiasco or Bond-CDS Negative Basis or How to Lose a Billion Dollars on a Trade, it is like a teenage lucidity head rush.
Avellaneda – 2010 Quant of the Year posts regularly from his NYU faculty page and covers Research and market commentary, Stochastic Calculus, PDEs for Finance, Risk and Portfolio Management.
Bookstaber – Author of the book A Demon of Our Own Design, ran Firm Risk at Salomon back in the day, and now is Senior Policy Advisor at the SEC. See Physics Envy in Finance or Human Complexity: The Strategic game of ? and ?
DeLong – Even with the constant bitching about the press and Team Republican plus the liveblogging of World War 2, I have never seen a better EcoFin website, see DeLong and Summers: Fiscal Policy in a Depressed Economy or Econ 191: Spring 2012. DeLong’s blog really is the model for curation and commentary to a large audience.
Gowers – Rouse Ball chair, Cambridge U, Fields Medal 1998, see ICM 2010 or Finding Cantor’s proof that there are transcendental numbers, and he was piqued to comment Re: Steig Larsson, or perhaps the translator Reg Keeling in Wiles meets his match. So, Salander’s picture perfect memory, capacity to defeat armed motorcycle gangs in hand-to-hand combat, and assorted other superpowers pass without comment but she thinks she has a proof of Fermat, you gotta call a mathematician to check yourself before you wreck yourself. Gowers is on the Heroes list forever, check.
Kahan – doesn’t publish so much anymore but he is the Edgar Allen Poe of floating point computations gone wrong horror stories, and they are all here. He did IEEE 754 floating point standard and won a Turing Award. When and if he has something to say, I will probably want to listen, see How Java’s Floating-Point Hurts Everyone Everywhere and Desperately Needed Remedies for the Undebuggability of Large Floating-Point Computations in Science and Engineering.
Lipton has a gloriously unique perspective presented in Godel’s Lost Letter. He provides the descriptive narrative for algorithm complexity in a public conversation typically dominated by proofs and expositions of computational models. If algorithm complexity was professional sports, its kind of like Lipton figured out there should be color commentators broadcasting live from the game. Top posts include: Interdisciplinary Research – Challenges, The Letterman Top Ten list of why P = NP is impossible, and The Singularity Is Here In Chess; its John Madden, Dick Vitale, and Andres Cantor meet Kurt Godel, John von Neumann, and Andrey Kolmogorov in the best possible way.
Tufte is “the guy” for the visual display of quantitative information. He has been the guy at least since the early 1980s and does not really publish the same way as Gowers, Lipton, or Tao. Tufte kind of figured out his publication flow before the internet, so you buy his books and if you want to know what he is thinking about now, you go to his course. He has stuff on line, lots of it, for example see his notebooks, or about ET. The Tufte course attendance is sort of mandatory, not sure but I think that’s in Dodd-Frank Title VII, so just do it before they find out.
Blelloch, Introduction to Data Compression, draft, here. Lossless compression chapters more polished.
Tom’s Hardware, IBM Has a Trillion-Bit, Insane Bandwidth “Holey Optochip”, “This claim boils down to a parallel optical transceiver that is first to boast the capability of transferring one trillion bits (1 Tbps or about 116.4 GBps). According to IBM, the chip is about eight times faster than any parallel optical component that is available today…”
Thompson Reuters, HFT page, here.
DB, HFT, 7 Feb 2011, here.
JPM, HFT An Overview, Mar 2012, here.