Information Theory/Discrete Memoryless Sources - Revision history

Maintenance script: Join split $$ formula lines for correct rendering

2026-03-16T13:46:58Z

Join split $$ formula lines for correct rendering

← Older revision		Revision as of 15:46, 16 March 2026
Line 48:		Line 48:
	$\text{Example 1:}$		$\text{Example 1:}$
	For the symbol probabilities of a quaternary source applies:		For the symbol probabilities of a quaternary source applies:
	:$$p_{\rm A} = 0.4 \hspace{0.05cm},\hspace{0.2cm}p_{\rm B} = 0.3 \hspace{0.05cm},\hspace{0.2cm}p_{\rm C} = 0.2 \hspace{0.05cm},\hspace{0.2cm}		:$$p_{\rm A} = 0.4 \hspace{0.05cm},\hspace{0.2cm}p_{\rm B} = 0.3 \hspace{0.05cm},\hspace{0.2cm}p_{\rm C} = 0.2 \hspace{0.05cm},\hspace{0.2cm}p_{\rm D} = 0.1\hspace{0.05cm}.$$
	p_{\rm D} = 0.1\hspace{0.05cm}.$$
	For an infinitely long sequence  $(N \to \infty)$		For an infinitely long sequence  $(N \to \infty)$
	*the  [[Theory_of_Stochastic_Signals/From_Random_Experiment_to_Random_Variable#Bernoulli.27s_law_of_large_numbers\|»relative frequencies«]]  $h_{\rm A}$,  $h_{\rm B}$,  $h_{\rm C}$,  $h_{\rm D}$   ⇒   a-posteriori parameters		*the  [[Theory_of_Stochastic_Signals/From_Random_Experiment_to_Random_Variable#Bernoulli.27s_law_of_large_numbers\|»relative frequencies«]]  $h_{\rm A}$,  $h_{\rm B}$,  $h_{\rm C}$,  $h_{\rm D}$   ⇒   a-posteriori parameters
Line 65:		Line 64:
	However,  if you replace the symbols with numerical values,  for example  $\rm A \Rightarrow 1$,   $\rm B \Rightarrow 2$,   $\rm C \Rightarrow 3$,   $\rm D \Rightarrow 4$, then you will get after <br>     »time averaging«   ⇒   crossing line     or     »ensemble averaging«   ⇒   expected value formation		However,  if you replace the symbols with numerical values,  for example  $\rm A \Rightarrow 1$,   $\rm B \Rightarrow 2$,   $\rm C \Rightarrow 3$,   $\rm D \Rightarrow 4$, then you will get after <br>     »time averaging«   ⇒   crossing line     or     »ensemble averaging«   ⇒   expected value formation
	*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#First_order_moment_.E2.80.93_linear_mean_.E2.80.93_DC_component\|»linear mean«]]   ⇒   »first order moment«:		*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#First_order_moment_.E2.80.93_linear_mean_.E2.80.93_DC_component\|»linear mean«]]   ⇒   »first order moment«:
	:$$m_1 = \overline { q_{\nu} } = {\rm E} \big [ q_{\mu} \big ] = 0.4 \cdot 1 + 0.3 \cdot 2 + 0.2 \cdot 3 + 0.1 \cdot 4		:$$m_1 = \overline { q_{\nu} } = {\rm E} \big [ q_{\mu} \big ] = 0.4 \cdot 1 + 0.3 \cdot 2 + 0.2 \cdot 3 + 0.1 \cdot 4= 2 \hspace{0.05cm},$$
	= 2 \hspace{0.05cm},$$
	*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#Second_order_moment_.E2.80.93_power_.E2.80.93_variance_.E2.80.93_standard_deviation \|»second order moment«]]:		*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#Second_order_moment_.E2.80.93_power_.E2.80.93_variance_.E2.80.93_standard_deviation \|»second order moment«]]:
	:$$m_2 = \overline { q_{\nu}^{\hspace{0.05cm}2} } = {\rm E} \big [ q_{\mu}^{\hspace{0.05cm}2} \big ] = 0.4 \cdot 1^2 + 0.3 \cdot 2^2 + 0.2 \cdot 3^2 + 0.1 \cdot 4^2		:$$m_2 = \overline { q_{\nu}^{\hspace{0.05cm}2} } = {\rm E} \big [ q_{\mu}^{\hspace{0.05cm}2} \big ] = 0.4 \cdot 1^2 + 0.3 \cdot 2^2 + 0.2 \cdot 3^2 + 0.1 \cdot 4^2= 5 \hspace{0.05cm},$$
	= 5 \hspace{0.05cm},$$
	*for the  [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Some_common_central_moments\|»standard deviation«]]  according to the  »Theorem of Steiner«:		*for the  [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Some_common_central_moments\|»standard deviation«]]  according to the  »Theorem of Steiner«:
	:$$\sigma = \sqrt {m_2 - m_1^2} = \sqrt {5 - 2^2} = 1 \hspace{0.05cm}.$$}}		:$$\sigma = \sqrt {m_2 - m_1^2} = \sqrt {5 - 2^2} = 1 \hspace{0.05cm}.$$}}
Line 96:		Line 93:
	$\text{Definition:}$  The  »'''maximum average information content'''«   of a message source depends only on the symbol set size  $M$  and results in		$\text{Definition:}$  The  »'''maximum average information content'''«   of a message source depends only on the symbol set size  $M$  and results in

	:$$H_0 = {\rm log}\hspace{0.1cm}M = {\rm log}_2\hspace{0.1cm}M \hspace{0.15cm} {\rm (in \ “bit")}		:$$H_0 = {\rm log}\hspace{0.1cm}M = {\rm log}_2\hspace{0.1cm}M \hspace{0.15cm} {\rm (in \ “bit")}= {\rm ln}\hspace{0.1cm}M \hspace{0.15cm}\text {(in “nat")}= {\rm lg}\hspace{0.1cm}M \hspace{0.15cm}\text {(in “Hartley")}\hspace{0.05cm}.$$
	= {\rm ln}\hspace{0.1cm}M \hspace{0.15cm}\text {(in “nat")}
	= {\rm lg}\hspace{0.1cm}M \hspace{0.15cm}\text {(in “Hartley")}\hspace{0.05cm}.$$

	*Since  $H_0$  indicates the maximum value of the  [[Information_Theory/Discrete_Memoryless_Sources#Information_content_and_entropy\|$\text{entropy}$]]  $H$,  $H_\text{max}=H_0$  is also used in our tutorial as short notation. }}		*Since  $H_0$  indicates the maximum value of the  [[Information_Theory/Discrete_Memoryless_Sources#Information_content_and_entropy\|$\text{entropy}$]]  $H$,  $H_\text{max}=H_0$  is also used in our tutorial as short notation. }}
Line 108:		Line 103:
	*The relations mentioned above are fulfilled due to the following properties:		*The relations mentioned above are fulfilled due to the following properties:

	:$${\rm log}\hspace{0.1cm}1 = 0 \hspace{0.05cm},\hspace{0.2cm}		:$${\rm log}\hspace{0.1cm}1 = 0 \hspace{0.05cm},\hspace{0.2cm}{\rm log}\hspace{0.1cm}37 > {\rm log}\hspace{0.1cm}6 > {\rm log}\hspace{0.1cm}2\hspace{0.05cm},\hspace{0.2cm}{\rm log}\hspace{0.1cm}M^k = k \cdot {\rm log}\hspace{0.1cm}M \hspace{0.05cm}.$$
	{\rm log}\hspace{0.1cm}37 > {\rm log}\hspace{0.1cm}6 > {\rm log}\hspace{0.1cm}2\hspace{0.05cm},\hspace{0.2cm}
	{\rm log}\hspace{0.1cm}M^k = k \cdot {\rm log}\hspace{0.1cm}M \hspace{0.05cm}.$$

	* Usually we use the logarithm to the base  $2$   ⇒   »logarithm dualis«    $\rm (ld)$,  where the pseudo unit  "bit"  $($more precisely:  "bit/symbol"$)$  is then added:		* Usually we use the logarithm to the base  $2$   ⇒   »logarithm dualis«    $\rm (ld)$,  where the pseudo unit  "bit"  $($more precisely:  "bit/symbol"$)$  is then added:

	:$${\rm ld}\hspace{0.1cm}M = {\rm log_2}\hspace{0.1cm}M = \frac{{\rm lg}\hspace{0.1cm}M}{{\rm lg}\hspace{0.1cm}2}		:$${\rm ld}\hspace{0.1cm}M = {\rm log_2}\hspace{0.1cm}M = \frac{{\rm lg}\hspace{0.1cm}M}{{\rm lg}\hspace{0.1cm}2}= \frac{{\rm ln}\hspace{0.1cm}M}{{\rm ln}\hspace{0.1cm}2}\hspace{0.05cm}.$$
	= \frac{{\rm ln}\hspace{0.1cm}M}{{\rm ln}\hspace{0.1cm}2}
	\hspace{0.05cm}.$$

	*In addition, you can find in the literature some additional definitions, which are based on the natural logarithm  $\rm (ln)$  or the logarithm of the tens  $\rm (lg)$.		*In addition, you can find in the literature some additional definitions, which are based on the natural logarithm  $\rm (ln)$  or the logarithm of the tens  $\rm (lg)$.
Line 128:		Line 119:
	We now consider the »'''information content'''«  of the individual symbols, where we denote the  "logarithm dualis"  with  $\log_2$:		We now consider the »'''information content'''«  of the individual symbols, where we denote the  "logarithm dualis"  with  $\log_2$:

	:$$I_\mu = {\rm log_2}\hspace{0.1cm}\frac{1}{p_\mu}= -\hspace{0.05cm}{\rm log_2}\hspace{0.1cm}{p_\mu}		:$$I_\mu = {\rm log_2}\hspace{0.1cm}\frac{1}{p_\mu}= -\hspace{0.05cm}{\rm log_2}\hspace{0.1cm}{p_\mu}\hspace{0.5cm}{\rm (unit\hspace{-0.15cm}: \hspace{0.15cm}bit\hspace{0.15cm}or\hspace{0.15cm}bit/Symbol)}\hspace{0.05cm}.$$
	\hspace{0.5cm}{\rm (unit\hspace{-0.15cm}: \hspace{0.15cm}bit\hspace{0.15cm}or\hspace{0.15cm}bit/Symbol)}
	\hspace{0.05cm}.$$

	You can see:		You can see:
Line 157:		Line 146:
	$\text{Definition:}$  The  »'''entropy'''«  $H$  of a discrete source indicates the  »'''mean information content of all symbols'''«:		$\text{Definition:}$  The  »'''entropy'''«  $H$  of a discrete source indicates the  »'''mean information content of all symbols'''«:

	:$$H = \overline{I_\nu} = {\rm E}\hspace{0.01cm}[I_\mu] = \sum_{\mu = 1}^M p_{\mu} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{p_\mu}=		:$$H = \overline{I_\nu} = {\rm E}\hspace{0.01cm}[I_\mu] = \sum_{\mu = 1}^M p_{\mu} \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{p_\mu}=-\sum_{\mu = 1}^M p_{\mu} \cdot{\rm log_2}\hspace{0.1cm}{p_\mu} \hspace{0.5cm}\text{(unit: bit, more precisely: bit/symbol)}\hspace{0.05cm}.$$
	-\sum_{\mu = 1}^M p_{\mu} \cdot{\rm log_2}\hspace{0.1cm}{p_\mu} \hspace{0.5cm}\text{(unit: bit, more precisely: bit/symbol)}
	\hspace{0.05cm}.$$

	The overline marks again a time averaging and  $\rm E[\text{...}]$  an ensemble averaging.}}		The overline marks again a time averaging and  $\rm E[\text{...}]$  an ensemble averaging.}}
Line 178:		Line 165:
	For the entropy of this binary source applies:		For the entropy of this binary source applies:

	:$$H_{\rm bin} (p) = p \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p\hspace{0.1cm}} + (1-p) \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{1-p} \hspace{0.5cm}{\rm (unit\hspace{-0.15cm}: \hspace{0.15cm}bit\hspace{0.15cm}or\hspace{0.15cm}bit/symbol)}		:$$H_{\rm bin} (p) = p \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{\hspace{0.1cm}p\hspace{0.1cm}} + (1-p) \cdot {\rm log_2}\hspace{0.1cm}\frac{1}{1-p} \hspace{0.5cm}{\rm (unit\hspace{-0.15cm}: \hspace{0.15cm}bit\hspace{0.15cm}or\hspace{0.15cm}bit/symbol)}\hspace{0.05cm}.$$
	\hspace{0.05cm}.$$

	This function is called  $H_\text{bin}(p)$  the  »'''binary entropy function'''«.  The entropy of a source with a larger symbol set size  $M$  can often be expressed using  $H_\text{bin}(p)$ .		This function is called  $H_\text{bin}(p)$  the  »'''binary entropy function'''«.  The entropy of a source with a larger symbol set size  $M$  can often be expressed using  $H_\text{bin}(p)$ .
Line 197:		Line 183:

	*$H_\text{bin}(p)$  is always a  »concave function«,  since the second derivative after the parameter  $p$  is negative for all values of  $p$ :		*$H_\text{bin}(p)$  is always a  »concave function«,  since the second derivative after the parameter  $p$  is negative for all values of  $p$ :
	:$$\frac{ {\rm d}^2H_{\rm bin} (p)}{ {\rm d}\,p^2} = \frac{- 1}{ {\rm ln}(2) \cdot p \cdot (1-p)}< 0		:$$\frac{ {\rm d}^2H_{\rm bin} (p)}{ {\rm d}\,p^2} = \frac{- 1}{ {\rm ln}(2) \cdot p \cdot (1-p)}< 0\hspace{0.05cm}.$$}}
	\hspace{0.05cm}.$$}}

	==Non-binary sources==		==Non-binary sources==
Line 208:		Line 193:
	For numerical calculation, the detour via the decimal logarithm  $\lg \ x = {\rm log}_{10} \ x$  is often necessary, since the  "logarithm dualis"  $ {\rm log}_2 \ x$  is mostly not found on pocket calculators.		For numerical calculation, the detour via the decimal logarithm  $\lg \ x = {\rm log}_{10} \ x$  is often necessary, since the  "logarithm dualis"  $ {\rm log}_2 \ x$  is mostly not found on pocket calculators.

	:$$H_{\rm quat}=\frac{1}{{\rm lg}\hspace{0.1cm}2} \cdot \left [ 0.4 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0.4} + 0.3 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0. 3} + 0.2 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0.2} + 0.1 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0.1} \right ] = 1.845\,{\rm bit}		:$$H_{\rm quat}=\frac{1}{{\rm lg}\hspace{0.1cm}2} \cdot \left [ 0.4 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0.4} + 0.3 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0. 3} + 0.2 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0.2} + 0.1 \cdot {\rm lg}\hspace{0.1cm}\frac{1}{0.1} \right ] = 1.845\,{\rm bit}\hspace{0.05cm}.$$
	\hspace{0.05cm}.$$

	{{GraueBox\|TEXT=		{{GraueBox\|TEXT=

Maintenance script: Add German interlanguage link

2026-03-16T12:27:27Z

Add German interlanguage link

← Older revision		Revision as of 14:27, 16 March 2026
Line 262:		Line 262:

	{{Display}}		{{Display}}
			[[de:Informationstheorie/Gedächtnislose_Nachrichtenquellen]]

Guenter at 14:52, 9 January 2024

2024-01-09T14:52:25Z

← Older revision		Revision as of 16:52, 9 January 2024
Line 84:		Line 84:
	*Is  $M = 1$, then each individual attempt will yield the same result and therefore there is no uncertainty about the output.		*Is  $M = 1$, then each individual attempt will yield the same result and therefore there is no uncertainty about the output.

	*On the other hand, an observer learns about an experiment with  $M = 2$, for example the  "coin toss"  with the set of events  $\big \{\rm \boldsymbol{\rm Z}(ahl), \rm \boldsymbol{\rm W}(app) \big \}$  and the probabilities  $p_{\rm Z} = p_{\rm W} = 0. 5$, a gain in information.  The uncertainty regarding  $\rm Z$  resp.  $\rm W$  is resolved.		*On the other hand, an observer learns about an experiment with  $M = 2$, for example the  »coin toss«  with the set of events  $\big \{\rm \boldsymbol{\rm Z}(ahl), \rm \boldsymbol{\rm W}(app) \big \}$  and the probabilities  $p_{\rm Z} = p_{\rm W} = 0. 5$, a gain in information.  The uncertainty regarding  $\rm Z$  resp.  $\rm W$  is resolved.

	*In the experiment  »dice«  $(M = 6)$  and even more in  »roulette«  $(M = 37)$  the gained information is even more significant for the observer than in the  »coin toss«  when he learns which number was thrown or which ball fell.		*In the experiment  »dice«  $(M = 6)$  and even more in  »roulette«  $(M = 37)$  the gained information is even more significant for the observer than in the  »coin toss«  when he learns which number was thrown or which ball fell.

Guenter at 14:48, 9 January 2024

2024-01-09T14:48:50Z

← Older revision		Revision as of 16:48, 9 January 2024
Line 196:		Line 196:
	*For  $p = 0$  this results in  $H = 0$, since the symbol sequence  $\rm B \ B \ B \text{...}$  can be predicted with certainty   ⇒   symbol set size only  $M = 1$.  The same applies to  $p = 1$   ⇒   symbol sequence  $\rm A \ A \ A \text{...}$.		*For  $p = 0$  this results in  $H = 0$, since the symbol sequence  $\rm B \ B \ B \text{...}$  can be predicted with certainty   ⇒   symbol set size only  $M = 1$.  The same applies to  $p = 1$   ⇒   symbol sequence  $\rm A \ A \ A \text{...}$.

	*$H_\text{bin}(p)$  is always a  "concave function",  since the second derivative after the parameter  $p$  is negative for all values of  $p$ :		*$H_\text{bin}(p)$  is always a  »concave function«,  since the second derivative after the parameter  $p$  is negative for all values of  $p$ :
	:$$\frac{ {\rm d}^2H_{\rm bin} (p)}{ {\rm d}\,p^2} = \frac{- 1}{ {\rm ln}(2) \cdot p \cdot (1-p)}< 0		:$$\frac{ {\rm d}^2H_{\rm bin} (p)}{ {\rm d}\,p^2} = \frac{- 1}{ {\rm ln}(2) \cdot p \cdot (1-p)}< 0
	\hspace{0.05cm}.$$}}		\hspace{0.05cm}.$$}}

Guenter at 17:24, 1 January 2024

2024-01-01T17:24:08Z

Guenter at 13:56, 28 February 2023

2023-02-28T13:56:21Z

← Older revision		Revision as of 15:56, 28 February 2023
Line 211:		Line 211:
	$\text{Example 3:}$		$\text{Example 3:}$
	Now there are certain symmetries between the symbol probabilities:		Now there are certain symmetries between the symbol probabilities:
	[[File:~~EN_Inf_T_1_1_S5~~.png\|frame\|Entropy of binary source and quaternary source ~~'''KORREKTUR: ""'''~~]]		[[File:EN_Inf_T_1_1_S5_v3.png\|frame\|Entropy of binary source and quaternary source]]

	:$$p_{\rm A} = p_{\rm D} = p \hspace{0.05cm},\hspace{0.4cm}p_{\rm B} = p_{\rm C} = 0.5 - p \hspace{0.05cm},\hspace{0.3cm}{\rm with} \hspace{0.15cm}0 \le p \le 0.5 \hspace{0.05cm}.$$		:$$p_{\rm A} = p_{\rm D} = p \hspace{0.05cm},\hspace{0.4cm}p_{\rm B} = p_{\rm C} = 0.5 - p \hspace{0.05cm},\hspace{0.3cm}{\rm with} \hspace{0.15cm}0 \le p \le 0.5 \hspace{0.05cm}.$$

Guenter at 13:55, 28 February 2023

2023-02-28T13:55:32Z

← Older revision		Revision as of 15:55, 28 February 2023
Line 184:		Line 184:
	The figure shows the binary entropy function for the values  $0 ≤ p ≤ 1$  of the symbol probability of  $\rm A$  $($or also of  $\rm B)$.  You can see:		The figure shows the binary entropy function for the values  $0 ≤ p ≤ 1$  of the symbol probability of  $\rm A$  $($or also of  $\rm B)$.  You can see:

	[[File:~~EN_Inf_T_1_1_S4~~.png\|frame\|Binary entropy function as a function of  $p$ ~~'''KORREKTUR: ""'''~~\|right]]		[[File:EN_Inf_T_1_1_S5_v2.png\|frame\|Binary entropy function as a function of  $p$ \|right]]
	*The maximum value  $H_\text{max} = 1\; \rm bit$  results for  $p = 0.5$, thus for equally probable binary symbols.  Then   $\rm A$  and  $\rm B$  contribute the same amount to the entropy.		*The maximum value  $H_\text{max} = 1\; \rm bit$  results for  $p = 0.5$, thus for equally probable binary symbols.  Then   $\rm A$  and  $\rm B$  contribute the same amount to the entropy.

Guenter at 14:32, 14 February 2023

2023-02-14T14:32:49Z

Show changes

Hwang at 10:42, 13 February 2023

2023-02-13T10:42:56Z

← Older revision		Revision as of 12:42, 13 February 2023
Line 186:		Line 186:
	==Non-binary sources==		==Non-binary sources==
	<br>		<br>
	In the  [[Information_Theory/~~Sources with Memory~~#~~Model_and_Prerequisites~~\|"first section"]]  of this chapter we considered a quaternary message source  $(M = 4)$  with the symbol probabilities  $p_{\rm A} = 0. 4$,   $p_{\rm B} = 0.3$,   $p_{\rm C} = 0.2$  and  $ p_{\rm D} = 0.1$.  This source has the following entropy:		In the  [[Information_Theory/Discrete_Memoryless_Sources#Model_and_requirements\|"first section"]]  of this chapter we considered a quaternary message source  $(M = 4)$  with the symbol probabilities  $p_{\rm A} = 0. 4$,   $p_{\rm B} = 0.3$,   $p_{\rm C} = 0.2$  and  $ p_{\rm D} = 0.1$.  This source has the following entropy:

	:$$H_{\rm quat} = 0.4 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0.4} + 0.3 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0. 3} + 0.2 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0.2}+ 0.1 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0.1}.$$		:$$H_{\rm quat} = 0.4 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0.4} + 0.3 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0. 3} + 0.2 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0.2}+ 0.1 \cdot {\rm log}_2\hspace{0.1cm}\frac{1}{0.1}.$$

Hwang at 10:41, 13 February 2023

2023-02-13T10:41:32Z

← Older revision		Revision as of 12:41, 13 February 2023
Line 198:		Line 198:
	$\text{Example 3:}$		$\text{Example 3:}$
	Now there are certain symmetries between the symbol probabilities:		Now there are certain symmetries between the symbol probabilities:
	[[File:EN_Inf_T_1_1_S5.png\|frame\|Entropy of binary source and quaternary source]]		[[File:EN_Inf_T_1_1_S5.png\|frame\|Entropy of binary source and quaternary source '''KORREKTUR: ""''']]

	:$$p_{\rm A} = p_{\rm D} = p \hspace{0.05cm},\hspace{0.4cm}p_{\rm B} = p_{\rm C} = 0.5 - p \hspace{0.05cm},\hspace{0.3cm}{\rm with} \hspace{0.15cm}0 \le p \le 0.5 \hspace{0.05cm}.$$		:$$p_{\rm A} = p_{\rm D} = p \hspace{0.05cm},\hspace{0.4cm}p_{\rm B} = p_{\rm C} = 0.5 - p \hspace{0.05cm},\hspace{0.3cm}{\rm with} \hspace{0.15cm}0 \le p \le 0.5 \hspace{0.05cm}.$$

← Older revision		Revision as of 19:24, 1 January 2024
Line 8:		Line 8:
	== # OVERVIEW OF THE FIRST MAIN CHAPTER # ==		== # OVERVIEW OF THE FIRST MAIN CHAPTER # ==
	<br>		<br>
	This first chapter describes the calculation and the meaning of entropy.  According to the Shannonian information definition,  entropy is a measure of the mean uncertainty about the outcome of a statistical event or the uncertainty in the measurement of a stochastic quantity.  Somewhat casually expressed,  the entropy of a random quantity quantifies its "randomness".		This first chapter describes the calculation and the meaning of entropy.  According to the Shannonian information definition,  entropy is a measure of the mean uncertainty about the outcome of a statistical event or the uncertainty in the measurement of a stochastic quantity.  Somewhat casually expressed,  the entropy of a random quantity quantifies its  »randomness«.

	In detail are discussed:		In detail are discussed:
Line 24:		Line 24:
	<br>		<br>
	We consider a discrete message source  $\rm Q$, which gives a sequence  $ \langle q_ν \rangle$  of symbols.		We consider a discrete message source  $\rm Q$, which gives a sequence  $ \langle q_ν \rangle$  of symbols.
	*For the variable  $ν = 1$, ... , $N$, where  $N$  should be "sufficiently large".		*For the variable  $ν = 1$, ... , $N$, where  $N$  should be sufficiently large.
	*Each individual source symbol  $q_ν$  comes from a symbol set  $\{q_μ \}$  where  $μ = 1$, ... , $M$~~, where~~  $M$  denotes the symbol set size:
			*Each individual source symbol  $q_ν$  comes from a symbol set  $\{q_μ \}$  where  $μ = 1$, ... , $M$.  $M$  denotes the symbol set size:

	:$$q_{\nu} \in \left \{ q_{\mu} \right \}, \hspace{0.25cm}{\rm with}\hspace{0.25cm} \nu = 1, \hspace{0.05cm} \text{ ...}\hspace{0.05cm} , N\hspace{0.25cm}{\rm and}\hspace{0.25cm}\mu = 1,\hspace{0.05cm} \text{ ...}\hspace{0.05cm} , M \hspace{0.05cm}.$$		:$$q_{\nu} \in \left \{ q_{\mu} \right \}, \hspace{0.25cm}{\rm with}\hspace{0.25cm} \nu = 1, \hspace{0.05cm} \text{ ...}\hspace{0.05cm} , N\hspace{0.25cm}{\rm and}\hspace{0.25cm}\mu = 1,\hspace{0.05cm} \text{ ...}\hspace{0.05cm} , M \hspace{0.05cm}.$$

	The figure shows a quaternary message source  $(M = 4)$  with ~~the~~ alphabet  $\rm \{A, \ B, \ C, \ D\}$  and an exemplary sequence of length  $N = 100$.		The figure shows a quaternary message source  $(M = 4)$  with alphabet  $\rm \{A, \ B, \ C, \ D\}$  and an exemplary sequence of length  $N = 100$.

	[[File:EN_Inf_T_1_1_S1a.png\|frame\|Quaternary source]]		[[File:EN_Inf_T_1_1_S1a.png\|frame\|Quaternary source]]
Line 36:		Line 37:
	*The quaternary source is fully described by  $M = 4$  symbol probabilities  $p_μ$.  In general it applies:		*The quaternary source is fully described by  $M = 4$  symbol probabilities  $p_μ$.  In general it applies:
	:$$\sum_{\mu = 1}^M \hspace{0.1cm}p_{\mu} = 1 \hspace{0.05cm}.$$		:$$\sum_{\mu = 1}^M \hspace{0.1cm}p_{\mu} = 1 \hspace{0.05cm}.$$
	*The message source is memoryless, i.e., the individual sequence elements are  [[Theory_of_Stochastic_Signals/Statistical Dependence and Independence#General_definition_of_statistical_dependence\|"statistically independent of each other"]]:		*The message source is memoryless, i.e., the individual sequence elements are  [[Theory_of_Stochastic_Signals/Statistical Dependence and Independence#General_definition_of_statistical_dependence\|»statistically independent of each other«]]:
	:$${\rm Pr} \left (q_{\nu} = q_{\mu} \right ) = {\rm Pr} \left (q_{\nu} = q_{\mu} \hspace{0.03cm} \| \hspace{0.03cm} q_{\nu -1}, q_{\nu -2}, \hspace{0.05cm} \text{ ...}\hspace{0.05cm}\right ) \hspace{0.05cm}.$$		:$${\rm Pr} \left (q_{\nu} = q_{\mu} \right ) = {\rm Pr} \left (q_{\nu} = q_{\mu} \hspace{0.03cm} \| \hspace{0.03cm} q_{\nu -1}, q_{\nu -2}, \hspace{0.05cm} \text{ ...}\hspace{0.05cm}\right ) \hspace{0.05cm}.$$
	*Since the alphabet consists of symbols  (and not of random variables) , the specification of  [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments\|"expected values"]]  (linear mean, second moment, standard deviation, etc.)  is not possible here, but also not necessary from an information-theoretical point of view.		*Since the alphabet consists of symbols  $($and not of random variables$)$,  the specification of  [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments\|»expected values«]]  $($linear mean, second moment, standard deviation, etc.$)$  is not possible here,  but also not necessary from an information-theoretical point of view.


	These properties will now be illustrated with an example.		These properties will now be illustrated with an example.

	[[File:Inf_T_1_1_S1b_vers2.png\|right\|frame\|Relative frequencies as a function of  $N$]]		{{GraueBox\|TEXT=
	~~{{GraueBox\|TEXT=~~		[[File:Inf_T_1_1_S1b_vers2.png\|right\|frame\|Relative frequencies as a function of  $N$]]
	$\text{Example 1:}$		$\text{Example 1:}$
	For the symbol probabilities of a quaternary source applies:		For the symbol probabilities of a quaternary source applies:
Line 50:		Line 51:
	p_{\rm D} = 0.1\hspace{0.05cm}.$$		p_{\rm D} = 0.1\hspace{0.05cm}.$$
	For an infinitely long sequence  $(N \to \infty)$		For an infinitely long sequence  $(N \to \infty)$
	*the  [[Theory_of_Stochastic_Signals/From_Random_Experiment_to_Random_Variable#Bernoulli.27s_law_of_large_numbers\|"relative frequencies"]]  $h_{\rm A}$,  $h_{\rm B}$,  $h_{\rm C}$,  $h_{\rm D}$   ⇒   a-posteriori parameters		*the  [[Theory_of_Stochastic_Signals/From_Random_Experiment_to_Random_Variable#Bernoulli.27s_law_of_large_numbers\|»relative frequencies«]]  $h_{\rm A}$,  $h_{\rm B}$,  $h_{\rm C}$,  $h_{\rm D}$   ⇒   a-posteriori parameters
	*were identical to the  [[Theory_of_Stochastic_Signals/Some_Basic_Definitions#Event_and_event_probability\|"probabilities"]]  $p_{\rm A}$,  $p_{\rm B}$,  $p_{\rm C}$,  $p_{\rm D}$   ⇒   a-priori parameters.
			*were identical to the  [[Theory_of_Stochastic_Signals/Some_Basic_Definitions#Event_and_event_probability\|»probabilities«]]  $p_{\rm A}$,  $p_{\rm B}$,  $p_{\rm C}$,  $p_{\rm D}$   ⇒   a-priori parameters.


	With smaller  $N$  deviations may occur, as the adjacent table (result of a simulation) shows.		With smaller  $N$  deviations may occur, as the adjacent table  $($result of a simulation$)$  shows.

	*In the graphic above an exemplary sequence is shown with  $N = 100$  symbols.		*In the graphic above an exemplary sequence is shown with  $N = 100$  symbols.

	*Due to the set elements  $\rm A$,  $\rm B$,  $\rm C$  and  $\rm D$  no mean values can be given.		*Due to the set elements  $\rm A$,  $\rm B$,  $\rm C$  and  $\rm D$  no mean values can be given.


	However, if you replace the symbols with numerical values, for example  $\rm A \Rightarrow 1$,   $\rm B \Rightarrow 2$,   $\rm C \Rightarrow 3$,   $\rm D \Rightarrow 4$, then you will get after <br>     »time averaging«   ⇒   crossing line     or     »ensemble averaging«   ⇒   expected value formation		However,  if you replace the symbols with numerical values,  for example  $\rm A \Rightarrow 1$,   $\rm B \Rightarrow 2$,   $\rm C \Rightarrow 3$,   $\rm D \Rightarrow 4$, then you will get after <br>     »time averaging«   ⇒   crossing line     or     »ensemble averaging«   ⇒   expected value formation
	*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#First_order_moment_.E2.80.93_linear_mean_.E2.80.93_DC_component\|"linear mean"]]   ⇒   "first order moment":		*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#First_order_moment_.E2.80.93_linear_mean_.E2.80.93_DC_component\|»linear mean«]]   ⇒   »first order moment«:
	:$$m_1 = \overline { q_{\nu} } = {\rm E} \big [ q_{\mu} \big ] = 0.4 \cdot 1 + 0.3 \cdot 2 + 0.2 \cdot 3 + 0.1 \cdot 4		:$$m_1 = \overline { q_{\nu} } = {\rm E} \big [ q_{\mu} \big ] = 0.4 \cdot 1 + 0.3 \cdot 2 + 0.2 \cdot 3 + 0.1 \cdot 4
	= 2 \hspace{0.05cm},$$		= 2 \hspace{0.05cm},$$
	*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#Second_order_moment_.E2.80.93_power_.E2.80.93_variance_.E2.80.93_standard_deviation \|"second order moment"]]:		*for the  [[Theory_of_Stochastic_Signals/Moments_of_a_Discrete_Random_Variable#Second_order_moment_.E2.80.93_power_.E2.80.93_variance_.E2.80.93_standard_deviation \|»second order moment«]]:
	:$$m_2 = \overline { q_{\nu}^{\hspace{0.05cm}2} } = {\rm E} \big [ q_{\mu}^{\hspace{0.05cm}2} \big ] = 0.4 \cdot 1^2 + 0.3 \cdot 2^2 + 0.2 \cdot 3^2 + 0.1 \cdot 4^2		:$$m_2 = \overline { q_{\nu}^{\hspace{0.05cm}2} } = {\rm E} \big [ q_{\mu}^{\hspace{0.05cm}2} \big ] = 0.4 \cdot 1^2 + 0.3 \cdot 2^2 + 0.2 \cdot 3^2 + 0.1 \cdot 4^2
	= 5 \hspace{0.05cm},$$		= 5 \hspace{0.05cm},$$
	*for the  [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Some_common_central_moments\|"standard deviation"]]  according to the  »Theorem of Steiner«:		*for the  [[Theory_of_Stochastic_Signals/Expected_Values_and_Moments#Some_common_central_moments\|»standard deviation«]]  according to the  »Theorem of Steiner«:
	:$$\sigma = \sqrt {m_2 - m_1^2} = \sqrt {5 - 2^2} = 1 \hspace{0.05cm}.$$}}		:$$\sigma = \sqrt {m_2 - m_1^2} = \sqrt {5 - 2^2} = 1 \hspace{0.05cm}.$$}}