I guess I view things along the lines if you had a database of 10,000 poker hands with the money that exchanged hands on each one, even if you deleted 5000 of the hands you would still have a great sample of the process at work. If you had 3 companies recording the poker hands, there would be no value in tracking down the reason for the discrepancy on hand #7877 with all 3 companies had the players holding different cards.

I think most trading backtesting is fools gold because like in the above example you could test for what happens when a player gets a certain hand over a certain number of deals and if they win or lose but that tells you absolutely nothing about what is going to happen on the next hand in the future. A deck of cards is surely more deterministic than the markets but we tend to think in terms of the opposite.

All you really need to know for R is how to get data in and then transform it to the input the function you want to use takes.

Sites with Summaries

scraperwiki.com (The community collects free data from the web and converts it into CSV, etc.)
wikiposit (The author collects free data from the web and converts it into CSV, etc.)
(recommendation) Quandl (Successor of wikiposit: great full-text search, XLS, CSV, JSON export)
Matlab Trendy (The community collects free data from the web and converts it into plots)

Foreign Exchange, Forex, FX

https://www.www.forexite.com/ (Forex, 1 minute ticks), CSV download
ratedata.gaincapital.com (Forex, tick data), CSV download
Forex Trading | Trade Currency Online | Forex Broker | OANDA (Forex, daily), CSV download
TrueFX — Tick-By-Tick Real-Time And Historical Market Rates, Clean, Aggregated, Dealer Prices (Forex tick data) after login
https://www.dukascopy.com/ (Forex, ticks), CSV
fxtop.com/de/historische-wechselkurse.php?MA=1 (Forex, long histories), Table

Bitcoin

bitcoincharts.com/charts/mtgoxUSD (Daily, many exchanges), Table
https://www.quandl.com/markets/bitcoin (summary of exchanges)

Debt/Interest Rates/Futures

historical time series
EMMI - European Money Markets Institute | Home (Interest-Rates: “Euribor, Eonia”), Excel download
RBS Databank (Exchange Rate, Interest Rate, Consumer Price Inflation (CPI), Commodities: “Brent Spot, Henry Hub Spot, I P E Brent crude futures 1-pos, I P E Natural Gas Index, OPEC Fixing, US EIA Crude Imports, WTI Daily”), Excel download
European Central Bank Interest-Rates and Government Bonds, web tables and partly Excel download
(German) Börse Düsseldorf - Hier wird gehandelt (Bonds, daily), Website table only
(German ) Deutsche Bundesbank - Startseite German money and interest-rate statistics
MRCI's Free Historical Futures Prices (many different future contracts, daily snapshots only)

current
(German) Bondboard - Index (Bonds)
(German) https://www.westlbmarkets.net (2012-07-12: Just went offline, hopefully online again soon), Swaps, Caps, Floors, CSV download
https://www.boerse-stuttgart.de/ (web table only) yield curves of bonds per rating and maturity
interest-rate derivatives
(new in the list) erstegroup.com Caps, Floors and Collars in form of Warrents

Stock / Indices

historical time series
finance.yahoo.com/(Stocks, Indices), Excel download
Google Finance: Stock market quotes, news, currency conversions & more, partial Excel download, otherwise website table only
Börsenkurse, Aktien & Finanznachrichten - ARIVA.DE, (Stocks, Indeces), Excel download, German only.
NASDAQ Stock Market - Stock Quotes - Stock Exchange News - [AUTOLINK]NASDAQ[/AUTOLINK].com (Nasdaq), Excel download
mba.tuck.dartmouth.edu (portfolio data), text download
(German) finanzen.handelsblatt.com
Investopedia - Educating the world about finance, Website table only
NASDAQ Stock Market - Stock Quotes - Stock Exchange News - [AUTOLINK]NASDAQ[/AUTOLINK].com, Website table only, intraday, daily
https://www.spglobal.com/ratings/en/ (S&P 500, daily, yearly)
current
finance.yahoo.com (Stocks, Indices)
Bloomberg Business (Stocks, Indices)

Investment Fonds

Issuer
Union Investment (CSV)

Energy

EU https://www.epexspot.com/de/marktdaten/intraday-handel (German), Website table only
US GDF SUEZ Energy Resources | Commercial Electricity Provider | Historical Pricing Data Login CSV download

General Historical

(German) Börse und Kurse: Aktuelle Börsenkurse in Realtime abfragen - OnVista (Stocks, Indices, Warrants, ETFs …, daily Quotes) Website
IndexMundi - Country Facts World-wide econometric data and commodities
Measuring Worth - Relative Worth Calculators and Data Sets US, UK, Japan, China with interest rates, GDP, inflation, gold and silver prices, cool long term, Website table only
Historia.se - Historicalstatistics.org Many countries, especially Sweden, GDP, population and much more, links to other sources
(German) Informationen und Termine zum Oktoberfest in München | Oktoberfest.de - Die Website zur Wiesn (Price of beer at Octoberfest in Munich, 1817-current)

Economic data

World

IndexMundi - Country Facts World-wide econometric data
OECD.StatExtracts includes data and metadata for OECD countries and selected non-member economies.
Google Data e.g. Market Capitalization in GDP: Google Data on Marcet Cap in % of GDP

European Union

Eurostat
European Central Bank
https://www.destatis.de/ Statistisches Bundesamt

United Kingdom

https://www.gov.uk/government/statistics/announcements

United States

Federal Reserve Economic Data – FRED
Census.gov
U.S. Bureau of Labor Statistics
The United States Social Security Administration
U.S. Department of the Treasury
U.S. Securities and Exchange Commission | Homepage
Economagic: Economic Time Series Page (Web, Excel download after login)
Standard & Poor's (Shiller-House Price Index)
Data.gov Public data of all US government organizations

Live Cams

Börse Frankfurt: Aktien Kurse, Charts und Nachrichten (German Dax)

Buy Data:

(I did not try any of these, but they look interesting.)

https://www.cmegroup.com (derivative prices)
Stock Options Analysis and Trading Tools on I [AUTOLINK]Volatility[/AUTOLINK].com (Implied volatilities)
commodities.euronext.if5.com (Cocoa Future, Robusta Coffee Future, White Sugar Future), 15 min delay, XML
End of Day Stock Quote Data and Historical Stock Prices (End-of-Day Stock), Excel download and more
https://www.historicaloptiondata.com/ (Option price data)
Stock Options Analysis & Options Trading Strategy Tools | Livevol (Implied volatilities)