SD Card CLK stops working properly on STM32F4 with SDIO connections

Question

I have an SD Card socket connected to STM32F4 with the connections as follows:

The firmware uses the following codes to initialize the SDIO:

    /* SDIO init function */
static void MX_SDIO_SD_Init(void)
{

  hsd.Instance = SDIO;
  hsd.Init.ClockEdge = SDIO_CLOCK_EDGE_RISING;
  hsd.Init.ClockBypass = SDIO_CLOCK_BYPASS_DISABLE;
  hsd.Init.ClockPowerSave = SDIO_CLOCK_POWER_SAVE_ENABLE;
  hsd.Init.BusWide = SDIO_BUS_WIDE_1B;
  hsd.Init.HardwareFlowControl = SDIO_HARDWARE_FLOW_CONTROL_DISABLE;
  hsd.Init.ClockDiv = 1;
}

And here's the system clock configuration:

void SystemClock_Config_HighSpeed(void)
{

  RCC_OscInitTypeDef RCC_OscInitStruct;
  RCC_ClkInitTypeDef RCC_ClkInitStruct;
  RCC_PeriphCLKInitTypeDef PeriphClkInitStruct;

  __HAL_RCC_PWR_CLK_ENABLE();

  __HAL_PWR_VOLTAGESCALING_CONFIG(PWR_REGULATOR_VOLTAGE_SCALE1);

  RCC_OscInitStruct.OscillatorType = RCC_OSCILLATORTYPE_HSE|RCC_OSCILLATORTYPE_LSE;     
  
  RCC_OscInitStruct.HSEState = RCC_HSE_ON;
  
  RCC_OscInitStruct.LSEState = RCC_LSE_ON;
  
  RCC_OscInitStruct.PLL.PLLState = RCC_PLL_ON;
  RCC_OscInitStruct.PLL.PLLSource = RCC_PLLSOURCE_HSE;
  RCC_OscInitStruct.PLL.PLLM = 8;
  RCC_OscInitStruct.PLL.PLLN = 336;
  RCC_OscInitStruct.PLL.PLLP = RCC_PLLP_DIV2;
  RCC_OscInitStruct.PLL.PLLQ = 7;
  if (HAL_RCC_OscConfig(&RCC_OscInitStruct) != HAL_OK)
  {
    Error_Handler();
  }

  RCC_ClkInitStruct.ClockType = RCC_CLOCKTYPE_HCLK|RCC_CLOCKTYPE_SYSCLK
                              |RCC_CLOCKTYPE_PCLK1|RCC_CLOCKTYPE_PCLK2;
  RCC_ClkInitStruct.SYSCLKSource = RCC_SYSCLKSOURCE_PLLCLK;
  RCC_ClkInitStruct.AHBCLKDivider = RCC_SYSCLK_DIV1;
  RCC_ClkInitStruct.APB1CLKDivider = RCC_HCLK_DIV4;
  RCC_ClkInitStruct.APB2CLKDivider = RCC_HCLK_DIV2;
  if (HAL_RCC_ClockConfig(&RCC_ClkInitStruct, FLASH_LATENCY_5) != HAL_OK)
  {
    Error_Handler();
  }

  PeriphClkInitStruct.PeriphClockSelection = RCC_PERIPHCLK_I2S|RCC_PERIPHCLK_RTC;
  PeriphClkInitStruct.PLLI2S.PLLI2SN = 192;
  PeriphClkInitStruct.PLLI2S.PLLI2SR = 2;
  
  PeriphClkInitStruct.RTCClockSelection = RCC_RTCCLKSOURCE_LSE;
  
  if (HAL_RCCEx_PeriphCLKConfig(&PeriphClkInitStruct) != HAL_OK)
  {
    Error_Handler();
  }

  HAL_RCC_MCOConfig(RCC_MCO1, RCC_MCO1SOURCE_HSE, RCC_MCODIV_4);

  HAL_SYSTICK_Config(HAL_RCC_GetHCLKFreq()/1000);

  HAL_SYSTICK_CLKSourceConfig(SYSTICK_CLKSOURCE_HCLK);

  /* SysTick_IRQn interrupt configuration */
  HAL_NVIC_SetPriority(SysTick_IRQn, 0, 0);
}

Generally this circuit works well in the beginning, but after about 2-3 years, some of the board with this circuit stops being able to use the SD Card even though there is no apparent damage to the components or lines. We have made thousands of this board and it's been 2-3 year and we are beginning to see several of the board with this "SD Card not mountable" issue. When I measured a working circuit looking at the SDIO connected to the SD card lines, the CLK line was functioning, and as soon as Card Detect (CD) went low, the CMD started, and eventually, D0, D1, D2, and D3 started and continued for a while and the firmware would return as SD_CARD_OK. The CLK line has a 44ns on and 16ns off at 400 kHz frequency.

On a non-working circuit, the clock is also functioning once in a while, and it has the same pulse the 44ns on and 16ns off at 400 kHZ, but then it would just abruptly stop after a couple of milliseconds.

Interestingly, the CLK would switch to a 50% duty cycle square wave 3.6us on/off at 138.812 kHz if I inject the SD card and try mounting again. Power cycling also sets it to this 138.812 kHz and there would be no response on other lines.

Because of the inconsistent clock, I wonder whether there's something wrong with the oscillator. I already tried putting a new 8 MHz Oscillator in but that didn't seem to change the behavior. I haven't tried changing the 30pF loading capacitors (C38 & C35). Can ceramic capacitors go bad?

The bad circuit and the good circuit have identical firmware, and they are powered with a lab benchtop power supply during testing so I know it's not a low battery issue. I used the same FAT32 formatted 64GB SD card in both cases. The error in the stm32f4xx_hal_sd.c when debugging ends here:

  if((response_r1 & SD_OCR_ILLEGAL_CMD) == SD_OCR_ILLEGAL_CMD)
  {
    return(SD_ILLEGAL_CMD);
  }

It makes sense since the CMD is just garbage since the CLK is not even the right frequency. This makes me think that it's something related to the timing but I'm not sure what I should replace components-wise. The STM32F407's other functions, like connecting as USB HID, still work.

I have been going back and forth between the good circuits and bad circuits for a while now and maybe it's time to just replace the capacitors and the oscillators and see whether that would work.

Has anyone else had experience dealing with this problem?