Information symmetries in irreversible processes

Chaos

Volumn 21, Issue 3, 2011, Pages

  Ellison, Christopher J ^a   Mahoney, John R ^b   James, Ryan G ^a   Crutchfield, James P ^a,c   Reichardt, Jörg ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c SANTA FE INSTITUTE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 80053432387     PISSN: 10541500     EISSN: None     Source Type: Journal    
DOI: 10.1063/1.3637490     Document Type: Article

References (66)

1. Crutchfield J.P. Ellison C.J. Mahoney J.R. James R.G. Chaos 2010, 20(3):037105. 10.1063/1.3489888, and
2. Upper D.R. Theory and Algorithms for Hidden Markov Models and Generalized Hidden Markov Models. 1997, Ph.D thesis (University of California, Berkeley)
3. This is equivalent to index notation in the Python programming language.
4. Levin D.A. Peres Y. Wilmer E.L. Markov Chains and Mixing Times 2008, and 1st ed. (American Mathematical Society, Providence, RI)
5. Crutchfield J.P. Physica D 1994, 75:11. 10.1016/0167-2789(94)90273-9.
6. Kelly L.P. Reversibility and Stochastic Networks 1979, (Wiley, Chichester)
7. Ephraim Y. Merhav N. IEEE Trans. Info. Theory 2002, 48:1518. 10.1109/TIT.2002.1003838
8. Baum L.E. Petrie T. Ann. Math. Stat. 1966, 37:1554. 10.1214/aoms/1177699147
9. MacDonald I. Zucchini W. Hidden Markov and Other Discrete-valued Time Series 1997, and (Chapman & Hill, London)
10. In Ref. 9, it was shown that Poisson-valued, state-emitting hidden Markov models are reversible if their internal Markov chains are reversible. This result does not hold with edge-emitting hidden Markov models, as demonstrated by example.
11. A similar statement can be made of Markov chains since the process generated by a Markov chain is reversible if and only the Markov chain is in detailed balance.
12. Blackwell D. Koopmans L. Ann. Math. Stat. 1957, 28:1011. 10.1214/aoms/1177706802
13. Ito H. Amari S.-I. Kobayashi K. IEEE Info. Theory 1992, 38:324. 10.1109/18.119690
14. Balasubramanian V. 1993, A. I. Tech Report 1370, Massachusetts Institute of Technology, January.
15. Weiss B. Monastsh. Math. 1973, 77:462. 10.1007/BF01295322.
16. Cover T.M. Thomas J.A. Elements of Information Theory 2006, and 2nd ed. (Wiley-Interscience, New York)
17. Crutchfield J.P. Ellison C.J. Mahoney J.R. Phys. Rev. Lett. 2009, 103(9):094101. 10.1103/PhysRevLett.103.094101
18. Ellison C.J. Mahoney J.R. Crutchfield J.P. J. Stat Phys. 2009, 136(6):1005. 10.1007/s10955-009-9808-z
19. Yeung R.W. IEEE Trans. Info. Theory 1991, 37(3):466. 10.1109/18.79902.
20. Kawabata T. Yeung R.W. IEEE Trans. Info. Theory 1992, 38(3):1146. 10.1109/18.135658
21. Shannon C.E. Weaver W. The Mathematical Theory of Communication 1962, and (University of Illinois, Champaign-Urbana)
22. Keep in mind that, unless otherwise stated, these figures are not drawn to scale. For example, the entropy of the past H[X:] is infinite. Since the drawings are not scale, we use the term circle liberally. Despite this, the important relationships of the variables are preserved.
23. del Junco A. Rahe M. Proc. AMS 1979, 75:259. 10.1090/S0002-9939-1979-0532147-2
24. Crutchfield J.P. Packard N.H. Physica 1983, 7D:201. 10.1016/0167-2789(83)90127-6
25. Grassberger P. Int. J. Theory Phys. 1986, 25:907. 10.1007/BF00668821.
26. Bialek W. Nemenman I. Tishby N. Neural Comput. 2001, 13:2409. 10.1162/089976601753195969
27. Crutchfield J.P. Feldman D.P. Chaos: Interdiscip. J. Nonlinear-Science 2003, 13(l):25. 10.1063/1.1530990
28. Mahoney J.R. Ellison C.J. Crutchfield J.P. J. Phys. A: Math. Theory 2009, 42:362002. 10.1088/1751-8113/42/36/362002
29. Crutchfield J.P. Young K. Phys. Rev. Lett. 1989, 63:105. 10.1103/PhysRevLett.63.105
30. Shalizi C.R. Crutchfield J.P. J. Stat. Phys. 2001, 104:817. 10.1023/A:1010388907793
31. Coven E.M. Paul M.E. Israel. J. Math. 1975, 20:165. 10.1007/BF02757884
32. Löhr W. Ay N. Adv. Complex Syst. 2009, 12(2):169. 10.1142/S0219525909002143
33. μ=H[S].
34. Hopcroft J.E. Ullman J.D. Introduction to Automata Theory, Languages, and Computation 1979, and (Addison-Wesley, Reading)
35. Angluin D. J. ACM 1982, 29:741. 10.1145/322326.322334.
36. Pin J.-E. LATIN'92, Lecture Notes in Computer Science 1992, 583:401-416. in Springer, Berlin/Heidelberg
37. Lombardy S. Automata, Languages and Programming, Lecture Notes in Computer Science 2002, 2380:787-787. in Springer, Berlin/Heidelberg
38. Sempere J. Grammatical Inference: Algorithms and Applications, Lecture Notes in Computer Science 2006, 4201:354-355. in Springer, Berlin/Heidelberg
39. García P. Vázquez de Parga M. Cano A. López D. Theor. Comput. Sci. 2009, 410(47-49):4961. 10.1016/j.tcs.2009.07.009
40. James R.G. Mahoney J.R. Ellison C.J. Crutchfield J.P. Many roads to synchrony: Natural time scales and their algorithms. and arXiv:1010.5545 (unpublished).
41. +.
42. In this example, it is sufficient to consider length-1 futures, but we use length-2 futures in order to demonstrate the general technique. That is, the columns of the matrix representing the conditional distribution must be marginalized in order to obtain the transition probabilities of the ε-machine.
43. 1). Matrices A and B are in the local time perspective and, thus, their forms are directly comparable. Matrix C, in contrast, is in the global (lattice) perspective of Fig. 7 and requires index manipulation to see that the resultant dynamics are irreversible.
44. Note that since the forward ε-machine is finite, the process does have a finite reverse generator-namely, the time-reversed forward ε-machine. However, the minimality of the ε-machine, within the class of unifilar HMMs, ensures that this presentation can be smaller than the reverse ε-machine only if it is also nonunifilar.
45. -). See Appendix.
46. In Fig. we stress that some atoms may have zero measure. For example, every ε-machine with uniformly distributed transition probabilities is exactly synchronizing. Thus, the atom representing hidden Markov models with uniformly distributed transition probabilities that are simultaneously minimal unifilar and not exactly synchronizing is empty.
47. Reference 1 called this class weakly asymptotically synchronizing, but it turns out to be equivalent to (strongly) asymptotically synchronizing (Ref. 65).
48. Travers N. Crutchfield J.P. J. Stat. Phys. and (in press).
49. Travers N. Crutchfield J.P. J. Stat. Phys. and (in press).
50. Travers N. Crutchfield J.P. J. Stat. Phys. and (in preparation).
51. Johnson B.D. Crutchfield J.P. Ellison C.J. McTague C.S. Enumerating Finitary Processes. and arXiv:1011.0036 (unpublished).
52. Given the forward ε-machine of a process, one can construct its reverse ε-machine using the technique described in Ref. 18. From this construction, we learn how the forward and reverse causal states are related. However, if one is given only the reverse ε-machine, then this important information is lost and must be deduced again. The bidirectional machine is a presentation of the process that preserves this information.
53. Let A, B, C, and D be random variables such that A maps deterministically onto D. Further, suppose that A and B are independent given C. Then it follows that D and B are also independent given C.
54. One must choose only one action: prediction or retrodiction. The forward bidirectional machine allows one to make a prediction, while the reverse bidirectional machine allows one to make a retrodiction. Making a simultaneous prediction and retrodiction with each machine does not yield the correct joint probabilities over predicted and retrodicted symbols.
55. Ball R.C. Diakonova M. MacKay R.S. Adv. Complex Syst. 2010, 13(3):327. 10.1142/S021952591000258X
56. Mahoney J.R. Ellison C.J. James R.G. Crutchfield J.P. How hidden are hidden processes? A primer on crypticity and entropy convergence. Chaos 2011, 21:037112. 10.1063/1.3637502,
57. Abdallah S.A. Plumbley M.D. A Measure of statistical complexity based on predictive information. and ", arXiv.org:1012.1890 (unpublished).
58. Abdallah S.A. Plumbley M.D. 2010, and Technical Report C4DM-TR10-10, Centre for Digital Music, Queen Mary University of London.
59. James R.G. Ellison C.J. Crutchfield J.P. Anatomy of a bit: Information in a time series observation. Chaos 2011, 21:037109. 10.1063/1.3637494,
60. Lind D. Marcus B. An Introduction to Symbolic Dynamics and Coding 1995, and Cambridge University Press, New York)
61. Landauer R. J. Stat. Phys. 1989, 54(5/6):1509. 10.1007/BF01044731.
62. Jarzynski C. Phys. Rev. Lett. 1997, 78(14):2690. 10.1103/PhysRevLett.78.2690.
63. Crooks G.E. J. Stat. Phys. 1998, 90(5/6):1481. 10.1023/A:1023208217925.
64. Toyabe S. Sagawa T. Ueda M. Muneyuki E. Sano M. Nat. Phys. 2010, 6:988. 10.1038/nphys1821
65. Travers N. 2011, personal communication (February)
66. Crutchfield J.P. Shalizi C.R. Phys. Rev. E 1999, 59(l):275. 10.1103/PhysRevE.59.275