PROGMEM problem with my project - dynamic allocation error

Question

My project is and RTISR (Real Time Instant Signal Recognizer). I'm sampling a signal from PIN A0 in the Arduino, and I have 6 types of Signals which I compare with (using squared cross correlation to avoid using sqrt() function).

Any way, to avoid killing my SRAM I wanted to burn the data of the 6 known signals in the Flash memory. I wanted to put there the signal, its Mean value, and its variance, which I calculate in advanced (with a function).

there is the code:

#include <avr/pgmspace.h>
#include <math.h>

#define T1_PS_L 5 //Timer1_Prescalers_Length
#define MAIN_CLK_FREQ 16000000
#define TIMER1_REG_CAPACITY 65536

#define NUM_OF_SAMPLES 64
#define SAMPLING_FREQUENCY 100

#define ANALOG_INPUT_PIN A0
#define WAVE_1  1   //PORTB PIN 8
#define WAVE_2  2   //PORTB PIN 9
#define WAVE_3  4   //PORTB PIN 10
#define WAVE_4  8   //PORTB PIN 11
#define WAVE_5  16  //PORTB PIN 12
#define WAVE_6  32  //PORTB PIN 13
#define NUM_OF_ETALONS 6

const short timer1_prescalers[] PROGMEM = {1, 8, 64, 256, 1024};
void SetTimer1CTCforFreq(float freq);
long get_prescaler_timer(float freq, char *ref_pres_indx);

double CrossCorrelation(const double x[], volatile double y[], int len, int delay);
double Variance(double arr[], int len);
double Mean(double arr[], int len);

/*-------------- ETALON SIGNALS ---------------------*/
const double sinWave1[NUM_OF_SAMPLES] PROGMEM = {0.0, 0.0980171403295606, 0.19509032201612825, 0.29028467725446233,
                           0.3826834323650898, 0.47139673682599764, 0.5555702330196022, 0.6343932841636455,
                           0.7071067811865475, 0.773010453362737, 0.8314696123025452, 0.8819212643483549,
                           0.9238795325112867, 0.9569403357322089, 0.9807852804032304, 0.9951847266721968, 1.0,
                           0.9951847266721969, 0.9807852804032304, 0.9569403357322089, 0.9238795325112867,
                           0.881921264348355, 0.8314696123025455, 0.7730104533627371, 0.7071067811865476,
                           0.6343932841636455, 0.5555702330196022, 0.47139673682599786, 0.3826834323650899,
                           0.2902846772544624, 0.1950903220161286, 0.09801714032956083, 1.2246467991473532e-16,
                           -0.09801714032956059, -0.19509032201612836, -0.2902846772544621, -0.38268343236508967,
                           -0.47139673682599764, -0.555570233019602, -0.6343932841636453, -0.7071067811865475,
                           -0.7730104533627367, -0.8314696123025452, -0.8819212643483549, -0.9238795325112865,
                           -0.9569403357322088, -0.9807852804032303, -0.9951847266721969, -1.0, -0.9951847266721969,
                           -0.9807852804032304, -0.9569403357322089, -0.9238795325112866, -0.881921264348355,
                           -0.8314696123025455, -0.7730104533627369, -0.7071067811865477, -0.6343932841636459,
                           -0.5555702330196022, -0.4713967368259979, -0.3826834323650904, -0.2902846772544625,
                           -0.19509032201612872, -0.0980171403295605};
const double sinWave2[NUM_OF_SAMPLES] PROGMEM = {0.0, 0.19509032201612825, 0.3826834323650898, 0.5555702330196022,
                           0.7071067811865475, 0.8314696123025452, 0.9238795325112867, 0.9807852804032304,
                           1.0, 0.9807852804032304, 0.9238795325112867, 0.8314696123025455,
                           0.7071067811865476, 0.5555702330196022, 0.3826834323650899, 0.1950903220161286,
                           1.2246467991473532e-16, -0.19509032201612836, -0.38268343236508967,
                           -0.555570233019602, -0.7071067811865475, -0.8314696123025452,
                           -0.9238795325112865, -0.9807852804032303, -1.0, -0.9807852804032304,
                           -0.9238795325112866, -0.8314696123025455, -0.7071067811865477,
                           -0.5555702330196022, -0.3826834323650904, -0.19509032201612872,
                           -2.4492935982947064e-16, 0.19509032201612825, 0.38268343236508995,
                           0.5555702330196018, 0.7071067811865474, 0.8314696123025452,
                           0.9238795325112865, 0.9807852804032303, 1.0, 0.9807852804032307,
                           0.9238795325112867, 0.8314696123025456, 0.7071067811865483,
                           0.5555702330196023, 0.3826834323650905, 0.19509032201612797,
                           3.6739403974420594e-16, -0.19509032201612725, -0.38268343236508984,
                           -0.5555702330196017, -0.7071067811865479, -0.8314696123025451,
                           -0.9238795325112864, -0.9807852804032305, -1.0, -0.9807852804032307,
                           -0.9238795325112867, -0.8314696123025456, -0.7071067811865485,
                           -0.5555702330196024, -0.3826834323650906, -0.19509032201612808};
const double sinWave3[NUM_OF_SAMPLES] PROGMEM = {0.0, 0.29028467725446233, 0.5555702330196022, 0.773010453362737,
                           0.9238795325112867, 0.9951847266721968, 0.9807852804032304, 0.881921264348355,
                           0.7071067811865476, 0.47139673682599786, 0.1950903220161286,
                           -0.09801714032956059, -0.38268343236508967, -0.6343932841636453,
                           -0.8314696123025452, -0.9569403357322088, -1.0, -0.9569403357322089,
                           -0.8314696123025455, -0.6343932841636459, -0.3826834323650904,
                           -0.0980171403295605, 0.19509032201612825, 0.47139673682599753,
                           0.7071067811865474, 0.8819212643483548, 0.9807852804032303,
                           0.9951847266721969, 0.9238795325112867, 0.7730104533627375,
                           0.5555702330196023, 0.29028467725446344, 3.6739403974420594e-16,
                           -0.2902846772544628, -0.5555702330196017, -0.7730104533627371,
                           -0.9238795325112864, -0.9951847266721969, -0.9807852804032307,
                           -0.8819212643483552, -0.7071067811865485, -0.47139673682599814,
                           -0.19509032201612808, 0.09801714032955978, 0.3826834323650897,
                           0.6343932841636447, 0.831469612302545, 0.9569403357322085, 1.0,
                           0.9569403357322088, 0.8314696123025457, 0.6343932841636455,
                           0.3826834323650907, 0.09801714032956088, -0.195090322016127,
                           -0.47139673682599564, -0.7071067811865477, -0.8819212643483547,
                           -0.9807852804032301, -0.9951847266721968, -0.9238795325112868, 
                           -0.7730104533627378, -0.5555702330196041, -0.2902846772544621};
const double rectWave4[NUM_OF_SAMPLES] PROGMEM = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
                                    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
                                    -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
                                    -1, -1, -1, -1, -1};
const double rectWave5[NUM_OF_SAMPLES] PROGMEM = {-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
                                    -1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1,
                                    -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 1,
                                     1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
const double triWave6[NUM_OF_SAMPLES] PROGMEM = {0.0, 0.0625, 0.125, 0.1875, 0.25, 0.3125, 0.375, 0.4375, 0.5,
                                  0.5625, 0.625, 0.6875, 0.75, 0.8125, 0.875, 0.9375, 1.0, 0.9375,
                                  0.875, 0.8125, 0.75, 0.6875, 0.625, 0.5625, 0.5, 0.4375, 0.375,
                                  0.3125, 0.25, 0.1875, 0.125, 0.0625, 0.0, -0.0625, -0.125, -0.1875,
                                  -0.25, -0.3125, -0.375, -0.4375, -0.5, -0.5625, -0.625, -0.6875,
                                  -0.75, -0.8125, -0.875, -0.9375, -1.0, -0.9375, -0.875, -0.8125,
                                  -0.75, -0.6875, -0.625, -0.5625, -0.5, -0.4375, -0.375, -0.3125,
                                  -0.25, -0.1875, -0.125, -0.0625};

const double sinWaveMean1 PROGMEM = (const double)Mean((double*)sinWave1, NUM_OF_SAMPLES);
const double sinWaveMean2 PROGMEM = (const double)Mean((double*)sinWave2, NUM_OF_SAMPLES);
const double sinWaveMean3 PROGMEM = (const double)Mean((double*)sinWave3, NUM_OF_SAMPLES);
const double rectWaveMean4 PROGMEM = (const double)Mean((double*)rectWave4, NUM_OF_SAMPLES);
const double rectWaveMean5 PROGMEM = (const double)Mean((double*)rectWave5, NUM_OF_SAMPLES);
const double triWaveMean6 PROGMEM = (const double)Mean((double*)triWave6, NUM_OF_SAMPLES);

const double sinWaveVariance1 PROGMEM = Variance((double*)sinWave1, NUM_OF_SAMPLES);
const double sinWaveVariance2 PROGMEM = Variance((double*)sinWave2, NUM_OF_SAMPLES);
const double sinWaveVariance3 PROGMEM = Variance((double*)sinWave3, NUM_OF_SAMPLES);
const double rectWaveVariance4 PROGMEM = Variance((double*)rectWave4, NUM_OF_SAMPLES);
const double rectWaveVariance5 PROGMEM = Variance((double*)rectWave5, NUM_OF_SAMPLES);
const double triWaveVariance6 PROGMEM = Variance((double*)triWave6, NUM_OF_SAMPLES);

const double* const Etalon_Signals[NUM_OF_ETALONS] PROGMEM = {sinWave1, sinWave2, sinWave3, rectWave4, rectWave5, triWave6};
const double Etalon_Signals_Mean[NUM_OF_ETALONS] PROGMEM = {sinWaveMean1, sinWaveMean2, sinWaveMean3, rectWaveMean4, rectWaveMean5, triWaveMean6};
const double Etalon_Signals_Variance[NUM_OF_ETALONS] PROGMEM = {sinWaveVariance1, sinWaveVariance2, sinWaveVariance3, rectWaveVariance4, rectWaveVariance5, triWaveVariance6};
/*-------------- SAMPLED SIGNAL ---------------------*/
volatile double signal_samples[NUM_OF_SAMPLES];
volatile int idx = 0;
/*-------------- CORRELATION VARS -------------------*/
volatile int counter = 0;
volatile double correlation = 0, MostSimillarCorrVal = 0, LocalMaxCorr = 0, maxCorrWave = 0;
volatile double SignalSamplesVar = 0, yMean = 0;
double EtalonSignalVal, EtalonMeanVal, EtalonVarianceVal, SignalSamplesVarTemp;


void setup()
{
  SetTimer1CTCforFreq(SAMPLING_FREQUENCY);
  DDRB = WAVE_1 | WAVE_2 | WAVE_3 | WAVE_4 | WAVE_5 | WAVE_6;
  pinMode(A0, INPUT);
  Serial.begin(115200);
}

void loop()
{
  Serial.println("hi");
  
}

ISR(TIMER1_COMPA_vect)
{
  signal_samples[idx] = analogRead(ANALOG_INPUT_PIN);
  Serial.println(signal_samples[idx]);
  for(int k = 0; k < NUM_OF_ETALONS; k++)
  {
    EtalonMeanVal = pgm_read_float(&Etalon_Signals_Mean[k]);
    EtalonVarianceVal = pgm_read_float(&Etalon_Signals_Variance[k]);
    Serial.println(EtalonVarianceVal);
    Serial.println(EtalonMeanVal);

    for(int delay = -NUM_OF_SAMPLES; delay < NUM_OF_SAMPLES; delay++)
    {
      counter = 0;
      while(counter < NUM_OF_SAMPLES)
      {
        SignalSamplesVarTemp = (signal_samples[(idx + counter - delay)%NUM_OF_SAMPLES] - yMean) * (signal_samples[(idx + counter - delay)%NUM_OF_SAMPLES] - yMean);
        SignalSamplesVar += SignalSamplesVarTemp;
        correlation += EtalonVarianceVal*SignalSamplesVarTemp;
        counter++;
      }
      correlation /= SignalSamplesVar * EtalonVarianceVal;
      LocalMaxCorr = LocalMaxCorr < correlation ? correlation : LocalMaxCorr;
    }
    if(MostSimillarCorrVal < LocalMaxCorr)
    {
      MostSimillarCorrVal = LocalMaxCorr;
      maxCorrWave = 1<<(k+1); // num of wave is 1 shifted left k+1 times.
    }
  }
  PORTB = maxCorrWave;
  idx = (idx+1)%NUM_OF_SAMPLES;
}

/*
  x - first signal.
  y - second signal.
  len - length of both of signals (or the max{LEN_x, LEN_y}).
*/
double CrossCorrelation(double x[], double y[], int len, int delay)
{
  double corr = 0, xVar = Variance(x, len), SignalSamplesVar = 0;
  double xMean = Mean(x, len), yMean = Mean(y, len);
  
  for (int i = 0; i < len; i++)
  {
    if ((i - delay) >= 0 && (i - delay) < len)
    {
      SignalSamplesVar += (y[i - delay] - yMean) * (y[i - delay] - yMean);
      corr += (x[i] - xMean) * (y[i - delay] - yMean) * (x[i] - xMean) * (y[i - delay] - yMean);
    }
  }

  double denominator = SignalSamplesVar * xVar;
  if (!denominator)
    return 0;
  else
    corr /= denominator;
  return corr;
}

double Variance(double arr[], int len)
{
  double mean = Mean(arr, len);
  double variance = 0;
  for (int i = 0; i < len; i++)
    variance += (arr[i] - mean) * (arr[i] - mean);
  return variance;
}

double Mean(double arr[], int len)
{
  double sum = 0;
  for (int i = 0; i < len; i++)
    sum += arr[i];
  return (sum / len);
}



long get_prescaler_timer(float freq, char *ref_pres_indx)
{
  bool loop_runs = true;
  float count_f = 0;
  long count_l = 0;
  char index = 0;
  long first_count_to_work = 0;
  char first_idx_to_work = 0;

  for (index = 0; index < T1_PS_L && loop_runs; index++)
  {
    count_f = MAIN_CLK_FREQ / (freq * timer1_prescalers[index]) - 1;
    count_l = (long)count_f;
    loop_runs = ((count_f - count_l) != 0 || count_l > TIMER1_REG_CAPACITY);
    first_count_to_work += count_l * (count_f < TIMER1_REG_CAPACITY && first_count_to_work == 0);
    first_idx_to_work += index * (first_count_to_work != 0 && first_idx_to_work == 0);
  }

  *ref_pres_indx = index;
  if (loop_runs) // if we didnt finish the loop succesfuly.
  {
    *ref_pres_indx = first_idx_to_work;
    count_l = first_count_to_work;
  }
  return count_l;
}

void SetTimer1CTCforFreq(float freq)
{
  char mask = 1;
  char pres_indx = 0;
  cli();    //stop interrupts
  TCCR1A = 0; // set entire TCCR1A register to 0
  TCCR1B = 0; // same for TCCR1B
  TCNT1 = 0;  // initialize counter value to 0
  // set compare match register for 1hz increments
  OCR1A = get_prescaler_timer(freq, &pres_indx); // = (16*10^6) / (1[HZ]*1024[prescaler]) - 1 (must be <65536)
  // turn on CTC mode
  TCCR1B |= (1 << WGM12);
  // Set CS10 and CS12 bits for 1024 prescaler
  do
  {
    TCCR1B |= mask * (pres_indx % 2); // in datasheet, values can be 1 - 5.
    pres_indx /= 2;
    mask = mask << 1;
  } while (pres_indx);

  // enable timer compare interrupt
  TIMSK1 |= (1 << OCIE1A);
  sei(); //allow interrupts
}

I'm getting the Error: "variable 'Etalon_Signals_Mean' with dynamic initialization put into program memory area" and it marks the line:

const double Etalon_Signals_Mean[NUM_OF_ETALONS] PROGMEM = {sinWaveMean1, sinWaveMean2, sinWaveMean3, rectWaveMean4, rectWaveMean5, triWaveMean6};

which are somewhere at the beggining (line 116). I dont understand why. My guess is that the compiler sees the "Mean" values which are a result of a function, and functions are "dynamic". But those values are getting into a const variable, so I don't get it. Please help.

`Mean` and `Variance` are functions which need to be executed in order to produce a result. You can't have the result of these functions assigned to const variables at compile time! The compiler isn;t going to run these functions for you to get a result so that it can then build the result into your code. — brhans, Jul 19 '20 at 22:01

score 2 · Answer 1 · answered Oct 15 '20 at 12:03

My guess is that the compiler sees the "Mean" values which are a result of a function, and functions are "dynamic".

Correct.

But those values are getting into a const variable, so I don't get it.

Correct, and the compiler doesn't get it either. That's why it complains.

In C++, you can actually let the compiler run a function at compile time, by marking it constexpr:

// here                  should be const by the way
// |                                 |
// v                                 v
constexpr /* <----- */ double Mean(const double arr[], int len) {
    // put the code here; the compiler must be able to see it to run it
}

I'm not sure how this interacts with PROGMEM. I think that the language doesn't know the difference between PROGMEM pointers and non-PROGMEM pointers, which is why you can simply dereference them and get the wrong data. If the compiler doesn't know about PROGMEM, then it will assume that you can dereference these pointers and get correct data. The compiler definitely doesn't know how to execute pgm_read_word. For this reason, you might need separate CompileTimeMean and RuntimeMean functions.

A different load instruction (on a different bus) is required to read data from flash. Hence the need for `pgm_read_word`. `__attribute__((__progmem__))` skips moving it to sram during the linker script. — Jeroen3, Oct 15 '20 at 12:17
@Jeroen3 Yes, but I don't think the compiler knows know any of that. That's why it lets you dereference progmem pointers without using pgm_read_whatever, which reads from the corresponding addresses in SRAM instead. — user253751, Oct 15 '20 at 14:50

score 0 · Answer 2 · answered Jun 05 '22 at 12:55

Yes, as mentioned in the other answer, C can only infer so many things at compile time. As for solutions, simply precompile the results, and keep them updated either in the code, or as separate object linked in. The former is probably easiest: create a program to calculate and printf() the values, and you can even pretty-print it with C array formatting to drop in directly. The result can be dumped to file (calctable.exe > table.c) or pasted in, however you like.

You will of course need a C toolchain for local/native use, which isn't a bad idea to have on hand anyway. MSVC, Cygwin, x86/ARM GCC, etc. are all fine options, give or take how much support / fancy IDE / barebones compiler / Linux compatibility / etc. you want. (On Linux of course, GCC's already available.)

PROGMEM problem with my project - dynamic allocation error

2 Answers2